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Decision  Modeling  for  Smart  Climatology 

The  purpose  of  smart  climatology  is  to  provide  information  that  would  be  useful  in  operational  planning 
at  lead  times  of  two  to  four  weeks  or  greater.  Decisions  made  at  shorter  lead-times  are  generally  informed 
by  forecasts.  One  of  the  challenges  in  designing  a  new  climatology  data  system — to  include  generation, 
storage,  and  access — is  anticipating  how  the  data  might  be  used,  and  therefore  how  to  design  the  system 
to  provide  the  most  useful  data  in  their  most  useful  format.  This  report  uses  an  operations  research  / 
decision  analysis  (OR/DA)  approach  to  prototype  the  process  of  using  smart  climatology  data  in  planning, 
and  identify  challenges  and  key  features  of  decision-relevant  climatology  data  system. 

1  How  does  METOC  information  create  value? 

While  some  commercial  forecast  and  climatology  providers,  such  as  The  Weather  Channel,  profit  from 
the  entertainment  value  of  forecasts,  military  meteorology  and  oceanography  (METOC)  products  create 
value  by  influencing  operational  consequences. 1  Taking  this  perspective,  an  OR/DA  framework  can  be 
used  to  analyze  and  improve: 

1)  how  METOC  information  is  used  in  decision-making  (mission  planning  and  execution); 

2)  the  design  of  METOC  products  for  communication  to  operational  decision-makers;  and 

3)  the  operational  value  created  by  METOC  information. 

The  OR/DA  framework  makes  some  basic  assumptions  that  may  be  non-intuitive  to  the  METOC 
community. 

First,  information  can  only  improve  operational  consequences  if  it  affects  decisions  -  in  other  words, 
if  the  information  induces  an  operational  end-user  (hereafter,  commander)  to  alter  her  course  of  action 
from  what  it  would  have  been  without  information  (or  with  different  information)  and  that  alteration  leads 
to  better  consequences. 

There  is  no  qualitative  difference  between  METOC  forecasts  and  climatology.  Both  are  information 
sets  and  can  be  interpreted  as  conditioning  the  probability  distribution  of  future  METOC  events. 
Therefore,  the  treatment  of  forecasts  and  climatology  is  analogous  in  OR/DA  analysis. 

In  OR/DA  analysis,  all  forecasts  are  treated  as  probabilistic.  Even  if  a  deterministic  forecast  is  offered 
-  e.g.  “conditions  will  be  green  for  the  mission”  or  “winds  will  be  20  knots  from  the  south,  with  low 
turbulence”  -  the  commander  knows  that  this  is  not  guaranteed,  and  may  hedge  his  decisions  based  on  his 
confidence  in  the  forecast(er)  or  based  on  his  experience  with  the  predictability  of  the  situation. 
Climatology  data  therefore  conditions2  the  probability  distribution  of  future  METOC  conditions,  affecting 
the  assessed  (objectively  or  subjectively  measured)  likelihood  of  future  METOC  conditions. 

Operational  consequences  are  produced  by  the  interaction  of  commanders’  decisions  with  METOC 
events.  METOC  conditions  alone  do  not  determine  the  mission  consequences.  If  they  did,  then  it  would 
not  be  valuable  for  commanders  to  have  METOC  information  -  the  same  consequences  would  occur  if 
commanders  simply  waited  for  the  mission  and  METOC  events  to  unfold,  without  accessing  or  using 
METOC  products. 


1  Commercial  forecast  and  climatology  providers  do  not  provide  only  entertainment  value  -  they  also  produce 
products  and  services  that  create  value  by  influencing  decisions. 

2  The  word  “condition”  is  used  here  as  a  verb,  defined  as  “to  subject  to  something  as  a  condition;  to  make  dependent 
on  a  condition  to  be  fulfilled;  to  make  conditional  on,  upon ”  (from  Oxford  English  Dictionary  online, 
http://dictionary.oed.com/entrance.dtl).  used  in  probability  to  indicate  the  probability  of  an  event  given  (conditioned 
on)  the  occurrence  of  another  event. 


Therefore,  OR/DA  analyses  are  meaningful  within  a  given  decision  context,  or  scenario.  The  best 
policies  for  using  METOC  information  (decision  policies),  the  best  design  of  a  METOC  product,  and  the 
value  of  METOC  information  are  all  dependent  on  the  specifics  of  a  given  decision  context  -  or  mission 
scenario.  Policies,  product  design,  and  value-of- information  cannot  be  analyzed  without  a  model  of  how 
METOC  information  is  used. 

Decision  policies,  product  design,  and  information  value  are  best  analyzed  in  expectation.  The 

consequences  of  an  actual  mission,  with  a  particular  forecast  and  verification  should  not  be  used  in 
isolation  to  assess  the  value  of  a  METOC  product  or  to  plan  for  future  uses  of  the  product.  To  use  a  single 
event  in  analyzing  a  METOC  product  would  be  like  saying  that  someone  is  a  good  gambler  because  he 
won  $1M  on  the  slot  machine  his  first  day  in  Las  Vegas.  The  OR/DA  approach  analyzes  METOC 
information  and  decisions  ex  ante,  i.e.  before  the  prevailing  conditions  at  the  time  of  verification  become 
known. 

1.1  OR/DA  Modeling 

In  order  to  use  an  OR/DA  approach  to  using,  designing,  and  valuing  METOC  products,  the  following 
elements,  and  the  relationships  among  them  must  all  be  understood,  and  if  possible  modeled,  for  a 
particular  decision  context: 

•  METOC  information  -  including  both  forecasts  and  climatology,  if  applicable; 

•  decisions  made  using  METOC  information; 

•  METOC  outcomes  (events),  i.e.  conditions  that  actually  prevail  in  forecast  verification;  and 

•  mission  consequences.3 

A  few  of  the  elements  and  relationships  pose  particular  challenges  for  the  analyst,  such  as  the  statistical 
relationship(s)  among  climatology,  forecasts,  and  METOC  outcomes.  The  term  “outcome”  is  used 
hereafter  to  refer  to  the  METOC  events  and  conditions  that  actually  prevail,  but  cannot  be  perfectly 
predicted  in  advance.  Modeling  how  decisions  interact  with  METOC  conditions  to  produce  mission 
consequences  is  often  one  of  the  most  challenging  parts  of  the  problem,  for  a  number  of  reasons  discussed 
in  more  detail  in  Section  7,  and  requires  expertise  from  multiple  disciplines,  including  details  related  to  a 
mission-specific  decision  context. 

In  the  Smart  Climatology  project,  this  was  the  major  hurdle  to  OR/DA  modeling.  In  order  to  build  a 
useful  model,  it  is  necessary  to  be  able  to  communicate  with  an  operational  subject-matter  expert  or  end- 
user  who  can  assess,  at  least  qualitatively,  what  are  the  commander’s  available  courses,  what  are  the 
METOC  conditions  that  would  affect  mission  consequences,  and  the  mission  consequences  that  would  be 
produced  by  each  combination  of  course  of  action  and  METOC  outcome.  For  example,  in  modeling  a 
missile  exchange,  the  subject-matter  expert  would  need  to  be  able  to  indicate  at  a  minimum  the  effective 
range  of  each  available  missile  type  under  various  METOC  conditions,  and  the  number  of  strikes  each 
aircraft  and  vessel  can  sustain  before  being  incapacitated.  In  this  project,  a  Naval  War  College  teaching 
scenario  was  used  as  the  decision  context,  and  therefore  the  area  experts  were  not  actually  operators  who 
could  provide  assessments  of  the  necessary  parameters.  Moreover,  for  reasons  further  described  in 
Section  2,  performance  data  for  platforms  and  weapons  systems  under  different  METOC  conditions  are 
not  necessarily  available  through  other  routes. 

1.2  Measuring  Value  of  Forecasts 

Value-of-information  (VOI)  is  a  OR/DA  concept  that  is  central  to  this  report,  and  that  has  been  used  in 
the  meteorology  literature  to  frame  problems  about  the  best  use  of  climate  and  forecast  information,  and 


3  To  indicate  measurable  operational  outcomes  or  mission  results,  this  report  uses  the  term  consequences  to  refer  to 
all  relevant  results/benefits/costs  associated  with  the  conduct  and  outcome  of  an  operation. 
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to  generate  estimates  of  the  socioeconomic  value  of  forecasts.  VOI  measures  the  value  of  information, 
such  as  a  forecast  or  climatological  information,  in  a  particular  decision  context.  Although  the  value  may 
be  measured  in  units  of  dollars,  mission-specific  value  may  also  be  measured  in  units  specific  to  the 
decision  context.  VOI  is  a  specific  measure  of  type  3  goodness  of  a  forecast,  by  Murphy  (1993)’s 
definitions. 

VOI  is  measured  as  the  expected  difference  in  consequences  under  two  information  scenarios:  a  baseline 
(less  informative)  scenario,  and  an  improved  (more  informative)  scenario.  For  example,  the  baseline 
scenario  could  use  maximum  daily  temperature,  while  the  more- informative  scenario  includes  predicted 
hourly  temperature.  A  baseline  scenario  could  be  a  categorical  precipitation  forecast,  while  the  more- 
informative  scenario  is  a  probabilistic  precipitation  forecast.  An  example  we  will  explore  more  is  a 
baseline  where  a  single  forecast  is  provided  for  a  valid  period,  and  the  more-informative  scenario  includes 
climatology  followed  by  a  forecast  for  the  same  valid  period. 

Since  consequences  are  affected  by  decisions  (in  turn  affected  by  METOC  information),  and  by  METOC 
outcomes,  each  of  the  elements  and  the  relationships  among  them  must  be  modeled.  This  implies  that 
efforts  to  integrate  METOC  forecasts  in  decision-making  should  focus  around  a  particular  decision,  rather 
than  around  a  particular  METOC  phenomenon  or  model. 

VOI  for  METOC  products  may  be  used  as  a  justification  for  METOC  budgets,  as  they  provide  a 
quantitative  estimate  of  the  benefits  of  METOC  information.  The  primary  reasons  the  VOI  framework  is 
used  here  are: 

•  VOI  highlights  the  relationships  between  the  generation  of  METOC  information,  the  way  it  is  used, 
and  the  most  valuable  characteristics  of  that  information. 

•  VOI  analysis  includes  identifying  optimal  decision  rules  as  a  function  of  METOC  information  -  and 
therefore  can  help  identify  improvements  in  the  use  of  information. 

•  VOI  provides  a  mission-centric  means  to  measure  information  quality  that  can  be  used  as  a  yardstick 
for  evaluating  improvements  to  the  information-generation  system,  such  as  a  forecasting  process  or  a 
climatology  database. 

The  OR/DA  approach  to  measuring  value  contrasts  with  methods  that  attempt  to  measure  individuals’ 
willingness  to  pay  for  METOC  forecasts.  These  methods  include  survey-based  approaches  such  as 
contingent  valuation  and  methods  to  tease  out  revealed  preferences,  by  observing  behavior  including 
market  prices.4 

1.3  Special  challenges  for  climatology 

OR/DA  analyses  using  climatology  information  pose  special  challenges.  Decisions  that  may  be 
influenced  by  climatology  require  long  lead  times,  of  at  least  two  to  four  weeks  but  potentially  one  to  two 
years  or  more  -  otherwise,  the  more  precise  information  provided  by  METOC  forecasts  would  be  used. 
Although  training  and  planning  occur  with  much  longer  lead  times,  most  operational  decisions  are  not 
made  -  or  are  not  made  irrevocably  -  until  lead  times  of  a  few  hours  to  a  few  days. 

In  addition,  decisions  based  on  climatology  are  rarely  the  last  decisions  that  affect  METOC-dependent 
mission  consequences.  In  general,  for  any  specific  mission  scenario,  there  is  some  flexibility  at  shorter 
lead  times  to  adjust  the  mission  and  influence  METOC-dependent  mission  consequences.  Therefore,  the 
analysis  of  the  use  of  forecast  information  must  be  included  in  the  analysis  of  the  use  of  climatology 
information. 

Nevertheless,  climatology  provides  value  by  informing  the  long-lead-time  decisions  that  create 
constraints  for  the  later  decisions.  For  example,  if  we  do  not  plan,  at  a  long  lead-time,  to  have  a  particular 


4  Johnson  and  Holt  (1997)  give  an  accessible  description  of  other  approaches  to  measuring  forecast  value. 
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asset  available  in-theater,  then  in  mission  execution,  alternative  courses  of  action  are  limited  by  the  non¬ 
availability  of  that  asset.  Some  US  Navy  bases,  for  example,  set  conditions  of  readiness  (TCCOR)  on  the 
basis  of  climatology  -  Kadena  is  in  TCCOR  4  throughout  during  the  typhoon  season,  identified  based  on 
climatology.  When  the  base  is  in  TCCOR  4,  they  make  preliminary  preparations  to  facilitate  preparations 
required  once  a  storm  is  forecast. 

A  naval  scenario  is  used  here  to  illustrate  the  use  of  forecast  and  climatology  information  in  decision¬ 
making,  and  to  illustrate  the  analysis  and  measurement  of  their  value  in  improving  expected  mission 
consequences.  The  long-lead-time  decision  that  is  based  on  climatology  data  is  where  to  station  Blue 
(U.S.)  bombers,  that  may  later  be  called  upon  (with  a  short  lead-time)  to  lay  sea-mines  to  defend  a  Green 
(neutral)  beach  against  a  Red  (enemy)  amphibious  attack. 

2  Modeling  an  operational  scenario  using  OR/DA  approach 

An  OR/DA  approach  to  using,  modeling,  and  evaluating  METOC  information  -  whether  forecasts  or 
climatology  -  focuses  on  two  elements  that  are  usually  outside  the  scope  of  METOC  analyses,  but  must 
be  brought  into  Tier  III  of  Battlespace  on  Demand:  5 

1.  decisions;  and 

2.  consequences. 

The  interacting  systems  through  which  METOC  information  creates  value  in  a  one-time  (tactical) 
decision  context  are  depicted  in  Figure  la.  As  discussed  earlier,  METOC  conditions  (the  outcome) 
combine  with  decisions  and  subsequent  actions  to  determine  mission  consequences.  Decisions  are 
informed  by  METOC  information  -  in  Figure  la,  this  is  depicted  as  the  forecast.  The  forecast  is  related  to 
the  METOC  outcome  -  ideally,  it  would  predict  the  outcome  perfectly.  In  reality,  both  the  forecast  and 
the  outcome  are  related  to  the  evolution  of  the  atmosphere  and  oceans.  As  indicated  in  Figure  la,  the 
forecaster  receives  information  inputs  -  including  observations  and  model  output  -  that  are  affected  by 
the  past  and  current  state  of  the  atmosphere  and  oceans.  The  METOC  outcome  is  also  affected  by  past 
and  current  state  of  the  atmosphere  and  oceans.  Figure  lb  depicts  the  way  these  relationships  are 
simplified  for  modeling  in  OR/DA.  Not  all  elements  are  modeled;  instead,  the  relationship  between  the 
forecast  and  METOC  outcome  is  modeled  statistically. 

Some  basic  notation  introduced  below  and  summarized  in  Table  1  will  help  clarify  this  discussion. 

For  now,  the  term  “forecast”  will  be  used  to  refer  to  any  METOC  message  -  which,  in  an  OR/DA 
framework,  could  be  either  a  forecast  or  climatology.  If  the  forecast  is  accurate  or  skillful,  the  METOC 
outcome,  w,  will  be  non-independent  of  the  forecast  (p  .  In  Figure  la,  this  relationship  is  depicted  such 
that  the  physical  systems  that  determine  METOC  outcomes  also  influence  the  information  available  to  the 
forecaster  and  thereby  the  forecast. 

In  OR/DA  modeling,  the  physical  systems  (atmosphere,  oceans)  are  not  modeled  directly.  Rather,  the 
dependence  of  both  the  METOC  outcome  and  the  forecast  on  the  physical  systems  are  modeled  by  a  joint 
probability  distribution  over  METOC  outcomes  and  forecasts,  as  depicted  in  Figure  lb. 

Sections  2. 1  through  2.4  below  describe  the  modeling  of  the  elements  and  relationships  depicted  in  Figure 
la.  Sections  2.5  and  2.6  describe  the  analysis  of  the  model,  including  calculation  of  the  value  of  METOC 
information.  Section  3  illustrates  this  modeling  and  analysis  process  using  a  decision  context  derived 
from  a  Naval  War  College  teaching  scenario  to  illustrate  the  modeling  process. 


5  The  three  tiers  are  described  in  Battlespace  on  Demand,  Naval  Oceanography  Program  Commander’s  Intent, 
released  November  2006. 
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affected  by  both 
choice  and  chance. 


Figure  lb:  Modeled  systems  that  produce  forecast 
value. 


Figure  la:  Interacting  systems  that  produce 
forecast  value. 


2.1  Alternative  Courses  of  Action 

The  first  two  steps  are  to  model  the  available  courses  of  action  (hereafter,  alternatives)  and  the  METOC 
conditions  that  will,  in  combination,  affect  operational  consequences.  The  decision  is  a  choice  of  exactly 
one  alternative  (a)  from  a  set  of  available  alternatives  (A).  In  any  operation  there  are  many  decisions 
that  are  made,  and  as  in  any  modeling  effort,  modeling  the  available  alternatives  involves  a  simplification 
of  the  real-world  situation.  One  of  the  major  simplifications  is  that  decisions  that  do  not  interact  with 
METOC  conditions  in  determining  consequences  may  be  neglected. 


Table  1:  Notation  for  single-stage  decision  modeling 


A 

the  set  of  all  alternatives  (courses  of  action)  available  in  a  given  decision  context 

a 

a  single  alternative,  a  £  A  (may  be  multi-dimensional) 

W 

the  set  of  all  possible  METOC  outcomes,  relevant  to  a  given  decision  context 

w 

a  single  METOC  outcome  that  may  prevail,  weJV  (may  be  multi-dimensional) 

O 

the  set  of  all  forecasts  that  are  possible,  ex  ante  (i.e.  before  the  forecaster  has  evaluated 
available  data,  and  generated  a  forecast)  in  a  given  decision  context 

(p 

a  single  METOC  forecast,  (may  be  multi-dimensional) 

In  a  simple,  one-state  decision  (as  described  in  Section  2),  cp  is  a  state  of  information. 

v(a,w) 

a  scalar  measure  of  the  consequences  of  the  selection  (and  implementation)  of  alternative 
a  and  the  occurrence  of  METOC  outcome  w ;  for  simplicity,  we  will  assume  that  lower 
values  of  v  are  preferred  -  i.e.  v  is  measured  as  costs,  or  negative  consequences 

a{cp) 

a  decision  rule  that  describes  the  alternative  the  commander  will  select  if  forecast  cp  is 
issued;  defined  V  cp  e  CD 

a{(p) 

the  normative  (optimal)  decision  rule;  V^>ecD,  a  [cp)=  minis'^  [v(a,w)]  . 

The  notation  E,  [  ]  means  the  expectation  taken  with  respect  to  the  random  variable  w , 
conditioned  on  the  forecast  (p  . 

the  set  of  all  possible  reference  forecasts;  if  the  reference  forecast  is  simply  climatology, 

=  =(pc ,  i.e.  there  is  only  one  possible  forecast 

Vr 

a  single  reference  forecast,  <pR  e  (may  be  multi-dimensional) 

2.2  ME  TOC  Outcomes 

METOC  outcomes  that  affect  consequences  of  concern  must  be  modeled,  as  must  the  functional 
relationship  between  METOC  outcomes  and  consequences  (discussed  in  Section  2.3). 

METOC  outcomes  that  affect  consequences  must  be  modeled  in  sufficient  detail  that: 

1.  the  probability  distribution  function  -  joint  over  all  relevant  variables  -  can  be  modeled,  for 
each  possible  state  of  information;6  and 

2.  consequences  as  a  function  of  each  decision-METOC  outcome  combination  can  be  modeled. 

Relevant  METOC  variables  are  those  that  will  significantly  affect  consequences  and/or  decisions. 
Identifying  the  set  of  relevant  METOC  variables  is  a  modeling  challenge,  and  should  be  done  in 
conjunction  with  modeling  consequences  and  decisions  (see  Sections  2.1  and  2.3). 


6  A  “state  of  information'’  is  one  of  the  possible  forecasts  or  climatology  data  draws,  or  a  single  realization  of  a 
forecast  or  data  draw.  If  there  are  multiple  states  -  either  decision  opportunities  or  forecast  opportunities  -  the  state 
of  information  will  describe  the  realizations  of  all  information  and  decision  opportunities  preceding  a  given  point. 
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2.3  Consequences 

To  indicate  measurable  operational  outcomes  or  mission  results,  this  report  uses  the  term  consequences  to 
refer  to  all  relevant  results/benefits/costs  associated  with  the  conduct  and  outcome  of  an  operation. 

When  METOC  conditions  are  operationally  relevant,  they  determine  consequences  in  combination  with 
decisions,  as  represented  in  Figure  2.  For  example,  the  damage  caused  to  an  air  base  by  a  typhoon  is 
determined  by  both  METOC  outcomes  -  i.e.  how  severe  the  typhoon  conditions  are  at  the  base  -  and 
decisions  -  i.e.  what  protective  actions  such  as  evacuation  of  aircraft  or  personnel,  protecting  buildings 
and  other  equipment  were  taken  before  the  storm  arrived. 

Therefore,  modeling  requires  measures  of  consequences  that  commanders  care  about,  as  a  function  of 
each  decision-METOC  outcome  combination.  Examples  of  mission-specific  consequences  include  lives 
lost,  targets  destroyed,  troops  (or  vehicles)  landing  successfully.  Consequences  may  also  be  measured  in 
terms  of  risk,  as  in  probability  that  Red  combat  power  exceeds  Blue  combat  power  or  probability  that  the 
mission  objective  is  achieved. 

To  be  complete,  consequences  should  include  METOC  impacts  plus  direct  consequences  of  decisions  - 
e.g.  protection  costs,  relocation  costs  for  bombers. 


Figure  2a:  METOC  outcomes  interact  with 
decisions  to  produce  consequences. 


Figure  2b:  Example  of  interaction  for  tropical 
cyclone  evacuation  decision. 


Modeling  decisions  (assumptions)  must  be  made,  to  generate  the  function  v(a,  w) ,  defined  in  Table  1.  If 
both  alternatives  A  and  METOC  outcomes  W  arc  discrete,  the  function  v(a,  vv)  can  be  represented 


with  a  decision  matrix  in  which  the  alternatives  form  the  rows,  and  the  METOC  outcomes  form  the 
columns,  and  the  cells  gives  the  consequences  of  each  decision-outcome  pair. 


Table  2:  Decision  matrix  example. 


w 

1 

2 

a 

1 

v(U) 

41.2) 

2 

42,1) 

v(2,2) 

2.4  METOC  Information 

METOC  information  -  whether  forecasts  or  climatology  -  must  also  be  modeled,  as  well  as  the 
relationship  between  information  and  outcomes.  Information’s  role  is  to  determine  probabilities  of 
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possible  METOC  outcomes.  In  other  words,  the  relationship  between  information  (whether  forecast  or 
climatology)  and  outcomes  must  be  modeled  probabilistically. 

A  probability  distribution  -  joint  over  all  relevant  variables  -  must  be  modeled,  conditional  on  each  state 
of  information  (possible  realization  of  the  forecast,  cp ).  Some  notation  given  in  Table  1  will  help  make 
this  discussion  more  precise. 

The  relationship  between  w  and  (p  is  described  by  a  joint  probability  density  function  (if  w  and  (p  are 
continuously  valued)  or  a  joint  probability  mass  function  (if  w  and  <p  take  on  only  discrete  values).  For 
simplicity,  both  pdfs  and  pmfs  are  referred  to  as  probability  distributions. 

•  Marginals:  The  notation  P\w\  or  P[w  =  z]  indicates  the  non-conditioned  (also  called  overall, 
marginal,  or  climatological)  probability  of  a  given  METOC  outcome  occurring.  The  notation 
PM  or  P\(p  =  y]  indicates  the  marginal  probability  of  a  given  forecast  being  issued. 

Forecasters  generally  do  not  think  of  a  forecast  as  a  random  variable  -  a  forecast  is  generated 
either  objectively  or  by  application  of  forecaster’s  knowledge.  However,  the  forecast  realization 
can  be  thought  of  as  random  before  observations  and  model  runs  that  determine  the  forecast  (or 
condition  the  generation  of  a  climatology)  are  generated.  At  this  point,  many  different  forecasts 
are  possible,  and  some  are  more  likely  than  others.  This  effect  is  modeled  by  the  probabilistic 
treatment  of  the  issuance  of  forecasts. 


•  Conditionals:  The  notation  P\w  \  <£>]  or  P[w  =  i  (p  =  y]  indicates  the  probability  of  a  given 
METOC  outcome  ( w  =  i )  occurring  given  that  a  particular  forecast  ( (p  =  j  )  has  been  issued 
(conditioned  on  a  particular  forecast).7 


Joint  probabilities:  The  notation  P\w,  (p\  or  P[w  =  i,(p  =  y]  indicates  the  probability  that  two 
events  will  both  occur:  a  given  forecast  will  be  issued  ( (p  =  j  )  and  a  given  METOC  outcome  will 


occur  ( w  —  i ).  This  means  that  P  [  vv,  <P\  <  P[  w\<p],  and  specifically  P\w\  cp\ 


P\w,(p\ 

P[(p] 


.This 


is  Bayes  Rule  (Clemen,  1997). 


Quantifying  the  statistical  relationship  between  METOC  information  and  METOC  outcomes  may  be 
difficult.  In  general,  measures  of  forecast  accuracy  and  skill  may  be  used  to  model  these  relationships. 
When  more  than  one  METOC  information  set  is  involved  -  as  when  both  climatological  information  and 
a  forecast  may  be  used  at  different  stages  in  planning  for  a  single  mission  -  the  relationships  between 
climatology  and  the  forecasts,  and  among  multiple  forecasts,  if  applicable,  must  also  be  modeled,  as  well 
as  their  statistical  relationship  to  METOC  outcomes.  Multiple  information  sets  are  addressed  in  Section  4. 


7  We  can  also  write  P\(p  |  w] ,  the  probability  that  forecast  (p  was  issued,  given  that  METOC  outcome  W  has 

occurred.  This  kind  of  conditional  is  less  common,  because  commanders  learn  the  forecast  chronologically  before 
the  METOC  outcomes  occur. 
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2.5  Decision  Policies 


How  information  will  affect  decisions  (and  thereby  actions)  must  be  modeled,  i.e.  the  decision  to  be  taken 
must  be  modeled  as  a  function  with  a  domain  of  every  possible  state  of  information.  In  the  notation  in 
Table  1,  the  possible  states  of  information  are  denoted  by  ® ,  and  a  single  possible  state  of  information  is 

cp  .  A  decision  policy  a  (<£>)  is  a  function  that  indicates,  for  each  cp ,  which  action  a  will  be  selected. 

A  decision  model  falls  into  one  of  two  major  categories:  normative  or  descriptive.  In  a  normative  model, 
the  assumption  is  that  the  decision  maker  optimizes,  by  minimizing  expected  total  cost.8  In  a  descriptive 
model,  the  decision  maker’s  behavior  may  not  be  optimal,  and  the  model  seeks  to  replicate  his  actual 
behavior  under  each  state  of  information.  Descriptive  modeling  is  therefore  much  more  difficult;  to 
develop  a  good  descriptive  model,  we  would  need  to  interview  commanders  and/or  observe  their  behavior 
to  try  to  understand  not  only  what  they  have  done  in  the  past,  but  what  they  would  do  under  hypothetical 

situations  -  because  we  must  know  a  (<£>)  \/(p  e  ®  . 

We  focus  on  normative  modeling  in  part  for  simplicity,  but  also  because  we  hope  to  induce  people  to 
behave  optimally  (normatively),  and  because  in  many  cases,  forecasters  are  implicitly  called  upon  to 
make  mission-related  decisions  -  by  issuing  a  particular  forecast,  they  may  effectively  force  commanders 
into  a  particular  course  of  action.  Therefore,  the  forecaster  should  understand  what  makes  a  decision 
optimal  in  the  decision  context  he  supports. 

The  role  of  OR/DA  analysts  is  to  model  decision  situations,  and  then  to  optimize  decisions,  often  under 
conditions  of  uncertainty.  To  optimize,  an  OR/DA  analyst  would  try  to  solve  the  following  problem,  for 
all  values  of  (p  : 


a  =mmE^[v(a,W)\  (1) 

The  solutions  to  Eq.  (1)  become  the  decision  rule,  as  indicated  in  the  definition  of  a  (<£>)  in  Table  1. 

This  problem  may  be  difficult  or  easy  to  solve,  depending  on  the  joint  probability  distribution  of  w  and 
(p ,  on  the  constraints  on  a ,  and  on  the  form  of  the  function  v(a,  vv) .  For  discrete  w , 

a*  (q>)  =  mm  [v(a,  tv)]  =  mm  £  v (a,  w) x  P[w  \  cp] . 

weW 

2. 6  Value  of  Information 

The  VOI  is  the  expected  difference  in  consequences  under  two  information  systems:  a  baseline  (less 
informative)  system,  and  an  improved  (more  informative)  system.  In  order  to  estimate  the  value  of 
information  provided  by  the  more  informative  system  -  for  example,  the  value  of  a  METOC  forecast  -  we 
must  model  a  baseline  or  reference  system,  which  we  will  call  a  reference  forecast. 

The  reference  forecast  could  be  a  forecast  that  is  less  accurate,  or  it  could  be  the  lack  of  a  forecast  -  in 
which  case,  climatological  frequencies  are  generally  used  as  the  reference  forecast.  If  the  reference 
forecast  is  simply  climatology,  =  ®c  =  <pR ,  i.e.  only  one  forecast  is  possible.  In  that  case,  (pR  is  a 

probability  distribution  across  all  METOC  outcomes,  and  specifically  Vw ,  P  [  vv  |  (pR  ]  =  the 
climatological  probability  (long-run  frequency)  of  METOC  outcome  w . 

The  modeled  alternatives,  METOC  outcomes,  and  consequences  do  not  change  when  a  reference  forecast 
is  introduced.  The  decision  policy,  however,  must  be  re-assessed  for  the  possible  values  of  the  reference 


8  Equivalently,  optimization  can  be  maximizing  expected  net  benefit  (costs  and  benefits  can  be  combined  by  treating 
costs  as  negative  benefits  and  vice  versa). 
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forecast.  The  identification  of  the  decision  policy  proceeds  exactly  as  described  in  Section  2.5, 
substituting  <pR  for  (p  .  In  other  words,  find  aR  ( (pK ) ,  V  cpR  e  O  R .  When  the  reference  forecast  is 

climatology,  there  is  only  one  possible  state  of  information,  therefore,  a  =  aRiK(pR^  is  a  constant.  Using  a 
normative  decision  policy  for  the  reference  forecast, 

al(P«)  =  ™™EMnlV(a’W)]- 


Once  the  decision  policy  is  identified,  the  expected  consequences  given  the  reference  forecast  is 


M  <Pr 


vte  w»w) 


The  VOI  is  defined  as  follows:9 


<Pr 


w\(pR 


v(aR{(pR),w) 


-E„ 


E, 


W\  (p 


>[a((p),w ) 


(2) 


3  Modeling  the  single-stage  Halsey  scenario 

A  model  of  force-on-force  interactions  has  been  developed  to  illustrate  how  METOC  information  could 
be  used  in  operational  planning  and  how  it  would  influence  execution  of  operations. 

The  model  describes  a  scenario  in  the  near  future  in  which  a  near-peer  competitor  (Red)  launches  an 
amphibious  invasion  on  a  neutral  (Green)  beach  in  East  Asia.  Blue  (U.S.  Navy)  defends,  while  Green 
does  not  participate. 

The  model  includes  operational  decisions  by  both  Blue  and  Red  that  are  made  on  the  basis  of  their 
respective  objectives  as  well  as  climatology  and  a  near-term  (72  hours  and  less)  METOC  forecast. 

Red  launches  an  unanticipated  amphibious  attack.  Red’s  effective  forces  include  both  amphibious  vessels 
and  escorts.  Blue  has  an  aircraft  carrier  in  the  region,  and  it  is  assumed  that  Blue  fighters  have  swept  the 
skies,  incapacitating  any  Red  air  power.  Moreover,  because  the  attack  is  launched  with  no  warning,  Red 
submarines  are  not  patrolling  close  enough  to  the  battle  to  reach  the  area  in  time  to  be  involved. 

The  engagement  therefore  consists  of  a  missile  exchange  between  Red’s  sea  forces  and  Blue’s  air  forces. 
The  consequences  of  missile  exchange  are  modeled  using  a  simple  salvo  model  following  Hughes  (1995). 
Later  (Section  5),  the  possibility  of  Blue  dropping  sea-mines  to  protect  the  beach  will  be  added. 

3.1  Alternative  Courses  of  Action 

In  the  single-stage  Halsey  scenario,  the  Blue  decision  is  a  choice  between  two  missile  types  for  an  air 
attack  on  Red  vessels.  The  missiles  differ  in  their  guidance  system  -  Type  A  is  METOC-sensitive,  and 
can  only  be  used  effectively  in  certain  METOC  conditions.  Each  fighter  can  carry  more  Type  A  than 
Type  B  missiles  in  a  single  sortie.  Therefore,  Type  A  would  be  preferred  when  METOC  conditions  allow 
its  use.  Type  B  uses  an  all-weather  GPS  guidance  system,  which  may  be  used  regardless  of  METOC 
conditions.  For  the  Halsey  scenario,  Blue’s  alternatives  are: 

a^A  ={A,B} 

where  a= A  if  Type  A  missiles  are  used,  and  a=  B  indicates  that  missile  of  Type  B  is  selected. 


1  This  formula  applies  when  v(a,  w)  is  in  units  of  cost  or  negative  consequences  (to  be  minimized).  If  v(a,  vv)  is 


in  units  of  benefit  (to  be  maximized),  then  VOI  =  E 


Jw\  (p 


>(a{(p),w) 


-E._ 


<Pr 


Jw\(pR 


v{aR((pR),w) 
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In  the  Halsey  scenario,  Red  also  has  a  decision  to  make  -  Red  escort  vessels  have  two  types  of  anti-air 
missiles,  C  and  D.  Red’s  Type  C  missiles  are  weather-sensitive,  and  in  each  salvo  the  escorts  can  fire 
more  rounds  of  Type  C  missiles  than  Type  D  missiles,  which  may  be  used  in  more  METOC  conditions. 

3.2  METOC  Outcomes 

In  the  Halsey  scenario,  the  relevant  METOC  outcomes  are  those  that  affect  the  ability  of  Red  and  Blue  to 
use  different  types  of  missile  guidance  systems  -  for  example,  less  expensive  (e.g.  radar  guided)  or 
longer-range  missiles  would  be  more  sensitive  to  atmospheric  conditions  and  sea  state  than  more 
expensive  (e.g.  GPS-guided)  or  shorter-range  missiles. 

For  simplicity,  we  will  model  METOC  outcomes  as  they  affect  the  usability  of  each  type  of  missile,  both 
Blue  and  Red.  In  other  words,  (F  =  {1,2,3,4|  where  w=l  indicates  that  all  missile  guidance  systems  can 
be  used,  w=  4  indicates  that  only  all-METOC  GPS  guidance  systems  can  be  used,  and  w  =2  or  3 
indicates  that  some  systems  can  be  used  and  others  cannot.  Table  3  indicates  the  effectiveness  of  the  four 
(two  Red  and  two  Blue)  missile  types. 

Outcomes  may  be  defined  by  multiple  METOC  parameters  -  for  example,  ceiling  and  visibility.  We  are 
therefore  making  the  assumption  that,  as  long  as  a  guidance  system  can  be  used,  its  effectiveness  is 
neither  reduced  nor  enhanced  by  METOC  conditions.  If  it  is  possible  to  estimate  the  probability 
distribution  over  these  definitions  of  METOC  outcomes,  and  there  are  no  further  operational  impacts 
caused  by  METOC  conditions,  then  METOC  outcomes  are  modeled  in  sufficient  detail. 

Table  3:  Relationship  between  METOC  outcomes  and  missile  use. 


Can  missile  be  used  in  these  METOC  conditions?  (Y/N) 


Missile  Type 

METOC  outcome  ( w ) 
12  3  4 

Blue  A 

Y 

Y 

N 

N 

B 

Y 

Y 

Y 

Y 

Red  C 

Y 

N 

N 

N 

D 

Y 

Y 

Y 

N 

3.3  Consequences 

In  the  missile-selection  problem,  the  consequences  of  concern  include  the  results  of  the  missile  battle  - 
the  number  of  Red  and  Blue  assets  destroyed  (or  surviving)  as  well  as  the  direct  cost  of  decisions  -  the 
expenditure  of  missiles.  Using  more  costly  missiles  implies  more  severe  consequences  than  using  less 
costly  missiles. 

For  modeling  purposes,  we  will  assume  that  the  costs  of  missiles  are  negligible  in  comparison  with  the 
results  of  the  engagement.  The  two  overriding  consequences  of  the  engagement  are  the  destruction  of  Red 
vessels  and  Blue  aircraft.  If  we  further  assume  that  the  defense  of  the  island  against  Red’s  amphibious 
attack  is  so  important  that  the  loss  of  Blue  aircraft  can  be  neglected,  and  the  commanders  will  focus  only 
on  the  destruction  of  Red  landing  capability,  then  we  can  define  consequences  as  the  number  of  Red 
landing  vessels  surviving  the  missile  attack.  To  make  the  consequence  measure  applicable  to  an  expanded 
scenario  that  includes  climatology  (treated  in  Section  5.3),  we  will  actually  define 

v(a,  tv)  =  number  of  Red  landing  vessels  surviving  to  land. 

Because  Red  will  be  able  to  directly  observe  METOC  conditions  before  choosing  missiles,  Red  will  use 
Type  C  missiles  whenever  that  will  lead  to  their  best  result,  and  otherwise  will  use  Type  D  missiles. 
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Therefore,  in  Sections  2  and  3  we  will  suppress  notation  for  this  choice.  In  Appendix  A,  Blue’s  missile 
choice  is  denoted  a2  and  Red’s  missile  choice  is  denoted  a3 . 

Clearly,  Blue  commanders  will  want  to  choose  a  e  A  to  minimize  v(a,  w) .  However,  neither  Blue  nor 
Red  can  control  w  .  The  task  at  hand  is  to  model  the  function  v(a,  vv) ,  which  can  then  be  represented  in 

a  decision  matrix,  with  two  rows  -  for  alternatives  A  ={A,B}  -  and  four  columns  -  for  METOC 
outcomes  IF={1,2,3,4}. 

The  modeling  of  v(a,  w)  requires  several  assumptions  and  is  based  on  the  basic  salvo  model  by  Hughes 
(1995),  pitting  Blue  aircraft  against  Red  vessels.  A  few  of  the  most  important  assumptions  include: 

•  Missiles  are  either  usable  or  unusable,  according  to  METOC  conditions,  as  detailed  in  Table  3.  If 
missiles  are  usable,  all  fired  missiles  are  effective. 

•  Red  has  two  vessel  types  -  escorts  and  landing  craft  -  and  Blue  does  not  distinguish  between 
them  in  targeting.  Parameters  such  as  number  of  missiles  fired  per  salvo  per  vessel  are  weighted 
averages  based  on  the  fraction  of  escort  vessels  among  total  Red  vessels,  and  assuming  no 
armament  on  the  landing  vessels. 

•  Fixed  number  of  missiles  fired  per  salvo  (which  corresponds  to  a  fighter  sortie)  per  aircraft/vessel 
until  aircraft/vessel  is  disabled;  thereafter,  it  neither  fires  nor  is  fired  upon. 

•  Each  aircraft/vessel  can  sustain  a  given  number  of  missile  strikes  before  becoming  disabled;  in 
each  salvo,  each  aircraft/vessel  can  defend  effectively  against  a  fixed  number  of  incoming 
missiles. 

The  assumptions  are  detailed  in  Appendix  A,  including  the  starting  number  of  aircraft  and  vessels  of  each 
type,  the  number  of  missiles  of  each  type  that  can  be  fired  per  salvo,  the  staying  power  and  defensive 
capability  of  each  platform. 

Table  4  shows  the  consequences  of  each  altemative-METOC  outcome  pair.  Small  values  are  desirable  for 
Blue,  because  they  indicate  a  smaller  Red  force  making  landing.  The  consequences  range  from  zero  Red 
landing  craft  surviving,10  to  200  landing  craft  surviving,  which  is  100%  of  the  Red  fleet. 

Table  4:  Decision  matrix:  Number  of  Red  landing  craft  remaining  after  missile  battle. 


v(a,  w) 

1 

w 

2 

3 

4 

a  B 

118 

0 

200 

200 

151 

75 

75 

75 

3.4  METOC  Outcomes  and  Information 

Any  information  that  will  be  received  before  Blue  commits  to  a  relevant  decision  must  be  modeled.  In 
this  case,  the  decision  is  the  choice  of  a  missile  type.  There  is  a  delay  while  missiles  must  be  assembled 
and  loaded  and  the  fighters  travel  from  the  aircraft  carrier  to  the  engagement  zone.  In  addition, 
observations  of  the  engagement  zone,  from  a  distance,  will  be  imperfect.  Therefore,  this  decision  must  be 
made  on  the  basis  of  a  forecast  of  METOC  conditions  in  the  engagement  zone. 

We  will  assume  that  the  missile-selection  decision  (described  in  more  detail  in  Section  3.5)  will  be  made 
on  the  basis  of  a  12-hour  forecast.  We  will  further  assume  that  the  forecast  is  a  categorical  forecast  of  the 


10  Note  that  the  METOC  outcome  determines  the  missile  type  that  Red  may  use,  therefore  in  METOC  conditions 
W=  2,  Red  will  have  to  use  its  Type  B  (less  effective)  missiles.  When  they  go  up  against  Blue’s  Type  A  missiles,  the 
entire  Red  fleet  is  destroyed. 
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METOC  outcomes  defined  in  Table  4  above,  and  describes  the  performance  of  both  Red  and  Blue 
missiles.  In  other  words,  the  forecast  cp  =1  indicates  that  conditions  corresponding  to  w=\  (all  missile 
guidance  systems  are  usable)  are  anticipated.  Because  the  forecast  is  imperfect,  the  issuance  of  the 
forecast  cp=  1  does  not  necessarily  imply  that  w  =  1  will  occur.  Instead,  the  relationship  is  described  by  a 

joint  probability  distribution  (  /•’[w,  <p] ),  given  in  Table  5,  or  by  conditional  (  /-’[tv  \(p\)  and  marginal 

( P\(p\ )  distributions  that  can  be  derived  from  the  joint  distribution.  The  values  given  below  are 

illustrative  only;  in  a  real  problem,  they  would  be  modeled  based  on  historical  observations  and  forecast 
accuracy  measures. 

Table  5:  Conditional,  joint,  and  marginal  probability  distributions  of  forecasts  and  METOC  outcomes. 


p[w,(p\ 

1 

w 

2 

3 

4 

1 

0.32 

0.07 

0.05 

0.02 

2 

0.03 

0.14 

0.02 

0.01 

3 

0.01 

0.02 

0.12 

0.03 

4 

0.01 

0.02 

0.03 

0.12 

p[w\cp\ 

1 

W 

2 

3 

4 

P[(p\ 

1 

0.70 

0.15 

0.10 

0.05 

0.45 

2 

0.15 

0.70 

0.10 

0.05 

0.20 

3 

0.05 

0.10 

0.70 

0.15 

0.18 

4 

0.05 

0.10 

0.15 

0.70 

0.18 

3.5  Decision  Policies 

In  the  Halsey  scenario,  we  will  model  decisions  normatively.  Therefore,  we  want  to  choose,  for  each 
^eO,  the  action  a  e  A  that  minimizes  the  expected  number  of  Red  landing  vessels  surviving  the 
missile  battle.  We  calculate,  V  cp  e  <t>  ,  Va  e  A 

EM<p\_v(a’w)\=  Z  v(d,w)xP[w\(p\. 

weJV 


Next,  following  Eq.  (l),for  each  (p  ,  we  identify  the  action  a  that  produces  the  smaller  value  of 

Ew\ip  [v(a,  vvjj  .  Anticipating  that  the  commander  will  make  the  optimal  decision  on  receiving  forecast 

(p  ,  by  following  decision  rule  a  ),  we  can  calculate  the  expected  consequences  of  receiving  a  given 


forecast:  E, 


W\(p 


v(a  (cp),w) 


Table  6:  Decision  policy  calculations  for  Halsey  scenario. 

EM  ?[v(a’w)] 


a 

1  2 

a  -a  [cp) 

Ew\<p 

v(a*(<p),w ) 

1 

113 

124 

A 

113 

<P 

2 

48 

83 

A 

48 

3 

176 

68 

B 

68 

4 

176 

26 

B 

26 
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3. 6  Value  of  Information 

The  reference  forecast  is  climatology,  denoted  <pR .  Conditional  on  the  reference  forecast,  w  is  distributed 
according  to  its  marginal  distribution,  P\w\ ,  given  in  Table  5  on  page  20.  The  expected  consequences  of 
using  each  missile  type,  conditional  on  the  reference  forecast  is 
E,\n  [v(a,w)|  (pR]='£jv(a,w)xP[w\. 

we.W 

Calculating  this  for  each  a  £  A ,  Ew^  =  A,  =122  and  \_v(ci  =  B,w)J  =  89.  The  optimal 

decision,  therefore,  is  always  to  use  Type  B  missiles,  a  =  aR(K(pR)  =  B ,  yielding  an  expected  consequence  of 

89  Red  landing  vessels  surviving.  Since  there  is  only  one  reference  forecast,  it  is  not  necessary  to  take  the 
expectation  with  respect  to  the  possible  values  of  the  reference  forecast,  and 

e<?r  [e^r  [v(ai  M’w)]]=e^r  [vte  (^m]=89- 

However,  to  calculate  VOI,  it  is  still  necessary  to  calculate  E(f)  E^fp  v(^a  ,  the  ex  ante 

expected  consequences  of  using  the  informative  forecast  (p ,  i.e.  before  the  forecast  is  issued.  Because 
METOC  outcomes  are  discrete,  the  formula  is  calculated  as  follows: 

pMxZ(v(aM’w)xp[wM)  =11- 

(p& O  V  weW  J 

Following  Equation  (2),  the  VOI  is  89—77  =  12.  In  other  words,  having  the  forecast  reduces  the  expected 
number  of  Red  landing  vessels  that  will  survive  the  missile  battle  from  89  to  77,  and  the  information 
contained  in  the  forecast  allows  the  disabling  of  12  Red  landing  vessels,  in  expectation. 

This  does  not  necessarily  mean  that  in  every  possible  METOC  outcome,  the  number  of  Red  vessels 
surviving  will  be  decreased  because  of  the  availability  of  a  METOC  forecast.  In  the  absence  of  a  forecast, 
the  reference  forecast  will  lead  the  Blue  commander  to  choose  Type  B  missiles.  With  the  forecast,  the 
commander  will  choose  Type  B  missiles  only  if  the  forecast  is  cp  =  3  or  cp  =  4,  and  she  will  choose  Type 
A  missiles  if  the  forecast  is  (p  =  1  or  cp  -  2.  If  she  chooses  Type  A  missiles,  and  the  METOC  outcome  is 
w  =  1  or  w  =  2,  then  the  decision  is  retrospectively  the  better  decision. 

However,  if  Blue  chooses  Type  A  missiles,  and  the  METOC  outcome  is  w  =  3  or  w  =  4,  then 
retrospectively,  the  commander  will  wish  she  had  chosen  Type  B  missiles  (see  Table  4,  page  19). 
However,  the  probability  of  regretting  the  choice  of  Type  A  missiles11  is  small  enough  that,  on  balance  (in 
expectation),  it  is  better  to  use  the  forecast,  and  choose  Type  A  missiles  when  the  forecast  indicates  (p  =  1 
or  (p  =2,  leading  to  the  disabling  of  33  additional  Red  landing  vessels  if  w—  1  and  75  additional  Red 
landing  vessels  if  w  —  2. 

4  Using  and  Valuing  Climatology 

It  is  less  obvious  how  to  use  climatology  than  METOC  forecasts,  and  its  effects  are  harder  to  see. 
Forecasts  are  more  precise,  more  detailed,  more  geographically  specific,  and  more  accurate  than 
climatology.  This  section  describes  the  role  of  climatology  as  an  informational  input  to  operational 

11  This  probability  is  10%,  calculated  from  joint  distribution  in  Table  5  as  the  probability  that  (p  or  2  and  W=  1  or 
2. 
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planning,  and  how  can  influence  -  and  adds  can  value  to  -  operational  decision-making  and  mission 
outcomes. 

4.1  Adding  Climatology  to  Forecasts 

Decision  modeling  for  climatology  information  and  measuring  the  value  of  climatology  information 
poses  special  challenges.  Decisions  that  may  be  influenced  by  climatology  require  long  lead  times  - 
otherwise,  the  more  precise  information  provided  by  METOC  forecasts  would  be  used  to  inform  the 
decisions.  Moreover,  decisions  based  on  climatology  are  rarely  the  last  decisions  that  affect  METOC- 
sensitive  mission  consequences.  In  general,  for  any  specific  mission  scenario,  there  is  some  flexibility  at 
short  (less  than  10-day)  lead  times  to  adjust  the  mission  and  influence  METOC-dependent  mission 
consequences. 

Only  in  very  unusual  circumstances  would  climatology  be  the  sole  METOC  information  available  to 
commanders  planning  and  executing  operations.  The  current  METOC  enterprise  delivers  forecasts  - 
usually  detailed,  accurate,  and  mission-specific  -  at  critical  lead-times  for  mission  planning  and 
execution,  no  matter  how  small  the  mission.  However,  when  climatology  is  used  together  with  METOC 
forecasts,  climatology  adds  mission  value: 

Climatology  +  Forecasts  >  Forecasts  Alone 

Sections  2  and  3  describe  the  use  of  a  single  forecast  -  a  forecast  for  METOC  conditions  verifying  at  a 
single  time  (in  the  Halsey  scenario,  the  time  of  the  missile  exchange)  -  to  inform  a  single  decision  (Blue’s 
choice  of  missile  type).  Often,  however,  more  than  one  forecast  is  issued  for  a  single  verification  time  - 
for  example,  a  48-hour  forecast  may  be  issued,  and  the  next  day  at  24-hour  forecast  may  be  issued. 

Moreover,  climatology  may  be  available  weeks  or  months  in  advance  of  an  operation,  and  a  forecast, 
applicable  to  the  same  verification  time  and  place,  is  available  hours  or  days  before  the  operation.  Both 
are  information  sets  that  condition  the  probabilities  of  the  same  METOC  events. 

Even  when  a  forecast  for  the  relevant  METOC  outcomes  will  be  available  before  last  operational 
decisions,  climatology  can  add  operational  value  in  two  ways: 

1 .  Climatology  conditions  probability  distributions  for  later  METOC  events,  so  that  commanders 
may  be  working  with  sharper  and/or  more  accurate  probability  distributions  weeks  or  months  in 
advance  of  the  operation  than  if  they  did  not  have  climatology. 

2.  Decisions  that  are  made  with  long  lead-times  that  may  be  informed  by  climatology  create  the 
constraints  for  later  decisions  that  are  made  on  the  basis  of  forecasts.  For  example,  if  we  do  not 
plan,  at  a  long  lead-time,  to  have  a  particular  asset  available  in-theater,  then  in  mission  execution, 
alternative  courses  of  action  are  limited  by  the  non-availability  of  that  asset. 

4.2  Forecasts  and  Climatology  as  Information  Sets 

The  term  “information  set”  may  be  applied  to  either  climatology  or  to  a  METOC  forecast.  The  treatment 
of  forecasts  and  climatology  is  analogous  in  OR/DA  analysis.  The  only  difference  is  that  decisions  made 
using  climatology  information  are  usually  followed  by  later  forecasts  for  the  same  valid  period. 

Climatology  and  forecasts  may  be  presented  in  different  formats.  For  example,  a  forecast  may  be  issued 
probabilistically,  but  often  it  is  issued  as  a  snapshot  of  near- future  METOC  conditions,  sometimes  as  a 
deterministic  narrative  of  the  near-term  evolution  of  METOC  conditions  describing  the  anticipated 
behavior  of  a  METOC  system. 

Climatology  is  more  commonly  presented  as  a  mean,  and  perhaps  maximum  historical  value  of  a  METOC 
parameter;  sometimes  climatology  describes  frequency  of  specific,  usually  extreme  events,  such  as  a 
tropical  cyclone.  Climatology  information  is  rarely  expressed  in  terms  of  probabilities,  but  implicitly  is 
interpreted  as  conditioning  the  probabilities  of  METOC  outcomes.  For  example,  learning  that  significant 
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wave  heights  in  November  average  12  feet  makes  a  commander  assess  as  substantial  the  likelihood  of 
waves  of  12  feet  and  higher.  On  the  other  hand,  given  another  state  of  information  -  that  the  significant 
wave  heights  average  4  feet  -  the  commander  would  believe  waves  12  feet  and  higher  to  be  fairly 
unlikely,  and  act  accordingly. 

Regardless  of  whether  information  is  called  climatology  or  a  forecast,  it  serves  to  condition  the 
commander’s  probability  distribution  over  future  METOC  outcomes. 

This  interpretation  of  forecasts  and  climatology  is  unfamiliar  and  non-intuitive  to  many  in  the  METOC 
community.  Figure  3  represents  this  conditioning  effect  conceptually.  The  outer  oval  represents  all 
possible  future  states  of  the  ocean  and  atmosphere.  The  next  oval  represents  the  possibilities  after 
climatology  has  been  received,  indicating  a  particular  climatological  regime,  or  indicating  the  usual 
conditions  for  a  given  time  of  year.  Finally,  the  smallest  oval  indicates  the  possible  future  states 
remaining  after  a  forecast  has  been  issued.  The  forecast  provides  more  information  than  climatology,  by 
narrowing  the  subset  of  possibilities  further  -  but  still  within  the  set  of  possibilities  remaining  after  the 
climatology  occurred.  In  other  words,  climatology  narrowed  the  set  of  possibilities,  and  the  forecast 
narrowed  it  further.  In  Figure  3,  this  is  represented  as  only  a  subset  of  future  states  being  possible  -  it 
could  also  be  interpreted  as  reducing  the  probability  of  some  possible  future  states  and  increasing  the 
probability  of  others.  In  other  words,  the  probability  distribution  (rather  than  the  set  of  possibilities) 
narrows. 


prior 


climatology 


All  possible  future 
states  of 

ocean/atmosphere 

Future  states  of 
ocean/atmosphere 
that  are  possible, 
given  that  a  particular 
climate  mode  prevails 


forecast 


Future  states  of 
ocean/atmosphere 
that  are  possible, 
given  that  a  particular 
forecast  has  been 
issued 


Figure  3:  Conceptual  representation  of  conditioning  by  climatology  and  forecast  information 

4.3  Modeling  the  use  of  Climatology 

In  order  for  climatology  to  become  relevant,  the  following  must  be  added  to  the  modeled  scenario: 

•  The  climatology  information  set,  together  with  its  probabilistic  relationship  with  the  forecast 
information  set(s)  and  METOC  outcomes; 

•  One  or  more  decisions  that  must  be  made  irrevocably  between  the  receipt  of  climatology 
information  and  the  receipt  of  a  forecast  and  that: 
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o  Are  irreversible  (or  reversible  only  at  significant  cost), 
o  Restrict  the  range  of  possible  future  decisions,  OR 

o  Affect  the  costs,  effectiveness,  or  other  consequences  of  later  decisions  or  the 
consequences  of  METOC  outcomes. 

More  specifically,  at  each  stage  that  decisions  occur  (irrevocably),  the  information  set  and  its  probabilistic 
relationship  to  prior  and  succeeding  information  sets  must  be  modeled.  For  example,  climatology 
information  may  be  received,  and  then  later  forecast  information  may  be  received. 

To  proceed  with  OR/DA  analysis,  the  minimum  modeling  of  the  relationships  among  information  sets 
and  outcomes  is  (with  notation  as  defined  in  Table  7): 

•  P[w\(p{],  Vwe  W ,q)\  e  Oj ,  and  P\w  \  (p^\  Vive  W,(p2  e  ®2;  and 

•  ^W-V^efDj. 

The  above  can  assessed  separately  or  can  be  derived  from  a  complete  joint  probability  distribution  over 
(px,  (p2,  and  iv .  It  is  also  possible  for  the  two  information  sets  to  relate  to  different  METOC  outcomes  or 

outcome  variables,  such  that  iv;  and  Wt  would  be  modeled  for  multiple  METOC  events  indexed  by  i ;  the 
climatology  data  set  will  add  value  as  long  as  there  is  at  least  one  decision  made  that  meets  the 
requirements  bulleted  above. 
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Table  7:  Notation  for  modeling 

A 


multi-stage  decisions 

the  set  of  all  alternatives  (courses  of  action)  available  at  stage  i  in  the 
decision  process,  i  = 


a. 


the  alternative,  ai  e  At  selected  at  stage  i 


W 


the  set  of  all  possible  METOC  outcomes,  relevant  to  a  given  decision 
context 


w 


a  single  METOC  outcome  that  may  prevail,  w  e  W . 

w  may  be  multi-dimensional,  reflecting  multiple  METOC  variables 
and/or  METOC  conditions  at  multiple  decision-relevant  times  or 
geographic  locations. _ 


O 


the  set  of  all  states  of  information  possible  in  the  i'h  stage,  i  =  1  . 

Multiple  decisions  with  the  same  information  can  be  modeled  as  a  single 
choice,  and  likewise  multiple  informational  messages  received  without 
intervening  decision  opportunities  can  be  modeled  as  a  single  message. 
Therefore,  information  opportunities  are  matched  with  decision  stages. 

If  the  first  information  set  is  climatology,  O,  may  be  denoted  Oc 


% 


a  single  realization  of  the  state  of  information  at  the  i  decision  stage 
(equivalently,  the  i,h  information  opportunity),  <pt  e$(. 


a  scalar  measure  of  the  consequences  of  the  selection  (and 
implementation)  of  alternatives  ax  through  an  and  the  occurrence  of 
METOC  outcome  w 


a, 


or 


a  decision  rule  (policy)  that  describes  the  alternative  the  commander  will 
select  if  forecast  (pi  is  issued;  may  be  dependent  on  the  prior  alternatives 
selected. 

In  a  further  elaboration,  ai  (•)  may  be  modeled  as  dependent  on  METOC 
outcomes  in  some  dimensions.  In  this  case,  wx,...,Wj  occur  and  the 
commander  knows  their  outcomes  before  decision  a,  must  be  made. 


a 


The  normative  (optimal)  decision  rule;  in  general,  must  be  found  using 
backward  induction,  as  described  in  Section  4.4.  Therefore,  a  formula  is 
not  provided. _ 


The  set  of  all  possible  reference  forecasts  available  at  the  time  of  decision 
a, . 


(pR,i 


A  single  reference  forecast,  (pRi  e  <f)  R  ( . 
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The  decision(s)  made  before  receipt  of  forecast  information  require  modeling  -  at  least  two  decisions  (one 
made  with  only  climatology  information,  one  made  after  receipt  of  a  forecast)  must  be  modeled. 12 
Therefore,  at  least  two  alternative  sets  and  choices,  al  e  Ax ,  and  a2  e  A2  must  be  modeled.  The 
consequences  of  the  mission  must  be  extended  such  that  they  depend  on  all  METOC  outcomes  and  all 
decisions,  i.e.  v(al,a2,...,an,wl,w2,...,  wm)  . 

Section  5  will  illustrate  these  modeling  steps  using  the  Halsey  scenario,  and  analyze  the  scenario  to 
estimate  the  value  of  climatology  information.  The  remainder  of  Section  4  describes  the  analysis  of  a 
multistage  problem. 

4.4  Solving  for  normative  decision  policies 

In  a  multi-stage  decision  problem,  a  solution  consists  of  a  policy  that  indicates  the  alternative  selected  at 
every  decision  opportunity  and  for  every  possible  state  of  information.  For  example,  in  a  two-stage 

problem,  the  policy  should  tell  us  ax  {(px  =  l)  for  each  value  of  <px ,  and  a 2  for  every  possible 

combination  of  ax  and  (p2 . 13 

A  key  feature  of  multi-stage  decision  problems  is  that,  in  general,  it  is  impossible  to  determine  the  best 
decision  at  early  stages  without  considering  what  decisions  will  be  made  at  later  stages  -  because  the 
(expected)  consequences  of  early  decisions  depend  on  the  choices  that  will  be  made  later. 14 

The  tropical  cyclone  evacuation  decision  for  a  stationary  installation  is  a  classic  example  of  a  multi-stage 
decision:  a  sequence  of  updated  forecasts  provides  multiple,  interrelated  decision  opportunities.  The 
choices  made  at  early  decision  opportunities  affects  the  available  alternatives  later  -  for  example,  if  an 
evacuation  is  undertaken  early,  the  option  to  not  evacuate  is  no  longer  available.  There  is  generally  some 
flexibility  left  -  for  example  to  evacuate  a  short  distance  or  shelter-in-place  -  until  the  storm  is 
immediately  upon  the  location  in  question.  If  early  decisions  are  made  on  the  basis  of  early  (and  therefore 
less  accurate)  METOC  forecasts,  there  will  be  a  tendency  to  undertake  more  costly  alternatives  than 
necessary.  On  the  other  hand,  if  commanders  can  wait  for  improved  forecasts,  they  can  undertake 
evacuations  only  if  later  forecasts  show  a  continued  or  increasing  threat.  Later  evacuations  may  be 
expedited  and  therefore  more  costly  or  less  thorough  evacuations,  but  there  will  be  fewer  total 
evacuations  (because  sometimes  a  TC  that  appears  threatening  early  on  will  not  affect  the  location),  and 
lower  total  costs.  This  concept  is  elaborated  in  Regnier  and  Harr  (2006). 

Solution  Concept 

How  do  we  find  the  optimal  policy?  The  basic  solution  concept  is  to  assume  the  commander  will  make 
the  right  decisions  in  the  future.  Solutions  are  reached  by  backward  induction,  starting  with  the  last 
event(s)  in  the  sequence,  and  working  backwards.  For  every  possible  decision  situation  -  every  possible 
forecast  and  prior  alternatives  selected  -  the  alternative  that  optimizes  expected  consequences  (minimizes 
total  cost)  is  selected  and  all  other  alternatives  are  eliminated.  Eliminated  alternatives  should  not 
influence  earlier  decisions.  Assuming  the  commander  will  always  make  the  right  decisions,  the  eliminated 
alternatives  will  not  be  selected.  Every  uncertain  event  takes  on  the  expected  total  cost  associated  with  the 


12  If  there  were  no  flexibility  to  make  any  decision  after  receipt  of  the  METOC  forecast,  then  the  forecast  would  add 
no  value. 

13  This  includes  combinations  in  which  ax  ^  CCX  fpx  )  .  As  discussed  in  this  section,  decision  stages  must  be 
analyzed  in  reverse  order  i  =  n,  fl  —  1, . . . ,  1 ,  and  therefore  when  analyzing  late-stage  decisions,  it  will  not  be  known 
which  values  of  ax,a2,...  will  be  optimal. 

14  The  Halsey  scenario  has  a  special  stmcture  such  that  it  is  possible  to  identify  the  best  first-stage  decision  without 
anticipating  the  decisions  that  will  be  made  later.  However,  this  is  not  generally  true. 
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possible  outcomes  of  its  realization,  where  expectation  is  taken  conditional  on  all  information  to  that 
point. 

Details  on  the  solution  algorithm  are  provided  in  the  Appendix  B. 

4. 5  Value  of  In formation 

The  value  of  information  is  the  difference  between  the  expected  consequences  if  the  commander  uses  the 
information  whose  value  is  being  analyzed,  and  the  expected  consequences  if  the  she  uses  less  or  no 
information.  As  discussed  in  Section  2.6  and  illustrated  in  Section  3.6,  calculating  the  VOI  therefore 
requires  modeling  the  less  informative  case,  which  provides  the  reference. 

When  multiple  information  sets  -  e.g.  climatology  and  a  forecast  -  are  involved  in  determining  decisions 
and  consequences,  several  reference  cases  could  be  considered.  A  reference  case  might  use  some  but  not 
all  of  the  information  sets  available  in  the  full-information  case,  e.g.  forecast  but  no  climatology  or  vice 
versa,  or  neither. 

The  purpose  of  this  report  is  to  show  how  climatology  may  be  used  in  operational  planning;  therefore,  the 
reference  cases  may  include  forecasts  but  exclude  climatology.  A  further  issue  is  semantic  -  often  the 
term  “climatology”  is  used  to  describe  a  less-informative  (reference)  case.  The  reason  is  that  when  a 
forecast  is  absent  a  decision-maker  might  use  a  long-run  average  (climatological)  description  of  METOC 
conditions  in  place  of  a  forecast.  However,  this  paper  uses  the  term  “climatology”  to  refer  to  an 
informative  message  giving  a  long-run  average  (or  other  statistical  parameter)  of  METOC  outcomes,  but 
conditioned  on  some  information  about  the  anticipated  METOC  events.  For  example,  climatology  might 
be  the  long-run  average  conditioned  on  the  anticipation  of  El  Nino  mode.  The  VOI  for  this  climatology 
would  be  the  difference  in  expected  outcomes  if  the  decision-maker  uses  the  ENSO-conditioned 
climatology  in  place  of  the  average  climatology  with  all  ENSO  modes  mixed  together. 

A  naive  analyst15  might  estimate  the  VOI  for  climatology  by  calculating  the  expected  total  consequences 
of  using  ENSO-conditioned  climatology  (the  more-informative  case)  with  the  expected  total 
consequences  of  using  non-conditioned  climatology  (the  less-informative  case)  without  modeling  the 
impact  of  later  forecasts.  This  would  generally  overstate  the  VOI  by  failing  to  account  for  commanders’ 
flexibility  to  adjust  to  improved  information  available  closer  to  the  time  of  the  eventual  operation. 
Although  this  would  yield  a  higher  estimate  of  the  VOI  for  climatology,  the  unrealistic  assumption  would 
undermine  confidence  in  the  analysis  and  therefore  fail  to  demonstrate  climatology’s  value.  In  estimating 
the  value  of  climatology,  the  definition  of  the  reference  case  is  very  important. 

5  Halsey  Scenario  with  Climatology 

This  section  extends  the  Halsey  scenario  described  in  Section  3  to  illustrate  the  use  and  value  of 
climatology.  As  discussed  above,  in  order  for  climatology  to  add  mission  value,  there  must  be  a  decision 
that  can  be  made  upon  receipt  of  climatology  that  is  not  reversible  later,  when  forecasts  become  available. 
In  this  case,  the  long-range  decision  that  cannot  be  reversed  within  the  time  horizon  allowed  by  a  forecast 
is  where  to  station  the  bomber  fleet. 

A  climatology  is  issued  three  months  before  the  engagement,  that  describes  the  mode  -  El  Nino,  La  Nina, 
or  neither  -  that  is  anticipated  to  prevail.  With  this  information  set  in  hand.  Blue  must  decide  where  to 
station  its  bomber  fleet.  By  the  time  the  preparations  for  the  engagement  begin,  it  will  be  too  late  to 
change  the  stationing  of  the  bombers;  they  may  be  flown  from  their  station  if  METOC  conditions  permit. 

It  is  assumed  that  the  rules  of  engagement  allow  mining,  but  only  once  hostilities  have  been  initiated. 

Blue  has  the  ability  to  lay  sea-mines  in  the  engagement  zone,  and  its  ability  to  do  so  depends  on  how 

15  An  analyst  trained  in  METOC  science  might  be  naive  in  this  sense  -  although  he  would  be  highly  expert  in  the 
physics  of  meteorology  and/or  oceanography,  he  would  not  trained  in  or  (in  general)  familiar  with  the  OR/DA 
approach  to  modeling  information. 
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close  its  bomber  fleet  (which  will  drop  the  mines)  is  to  the  engagement  zone,  as  well  as  the  METOC 
conditions  prevailing  at  the  bomber  station.  The  rest  of  the  scenario  proceeds  as  described  in  Section  3. 
Blue  will  have  the  missile  alternatives  described  in  Section  3.1  and  the  forecast  for  the  engagement  zone 
described  in  Section  3.4.  The  consequences  of  Blue’s  actions  and  Red’s  response  will  be  affected  by  the 
effectiveness  of  sea-mining.  The  stationing  of  the  bombers  -  determined  earlier  -  and  the  METOC 
conditions  prevailing  at  its  bomber  station  may  limit  the  effectiveness  of  mining  operations.  The 
expanded  scenario  is  illustrated  in  the  decision  tree  in  Figure  4,  and  described  in  Sections  5. land  5.4. 


missile  conditions  at  missile 
at  bomber  engagement  selection  engagement  selection 
bases  zone  zone 

Figure  4:  Decision  tree  for  expanded  Halsey  scenario,  from  Blue’s  perspective.  Squares  represent  decision 
opportunities,  while  circles  represent  uncertain  events.  From  Blue’s  perspective,  Red’s  decision  is  uncertain. 
Notation  is  given  in 
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Table  8. 

5.7  Courses  of  Action 

There  are  two  decision  points,  each  having  a  set  of  alternative  courses  of  action.  The  first  decision  point 
( a1  e  Al  must  be  selected)  comes  after  receipt  of  climatology,  and  when  the  decision  is  where  to  station 

the  bombers.  The  second  decision  point  (a2  e  A1 ,  formerly  a  €  A)  occurs  after  the  receipt  of  a  forecast 

for  the  engagement  zone.  The  choice  -  identical  to  that  described  in  Section  3.1  -  is  between  Types  A  and 
B  air-to-surface  missile.  Table  8  provides  notation. 
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Table  8:  Notation  for  extended  Halsey  scenario 


a\ 

The  alternative  selected  at  the  first  decision  opportunity,  reflecting  where  bombers 
will  be  stationed,  ax  e  Ax 

A 

The  set  of  all  alternative  bomber  stations  available  -  Alaksa  (A),  Kadena  (K),  and 
Guam(G),  i.e.  y4j  =  {A,K,G} 

a2 

The  alternative  selected  at  the  second  decision  opportunity,  reflecting  choice  of  air-to- 
surface  missile  type  for  engagement  (denoted  a  in  Section  3). 

A 

The  set  of  all  alternative  missile  types  available,  A,  ={A,B}  (formerly  denoted  A  ). 
Note  that  in  this  scenario,  the  available  alternative  set  does  not  depend  on  earlier 
decisions  or  outcomes,  i.e.  A1  is  not  a  function  of  a, ,  wx. 

wx 

The  set  of  all  possible  outcomes  of  the  first  METOC  event,  describing  conditions  at 
the  potential  bomber  stations,  and  specifically  whether  conditions  permit  bombers  to 
fly  from  each  station. 

The  outcome  of  the  first  event,  indicating  which  METOC  conditions  prevail  at  the 
time  that  Red  initiates  hostilities,  wxeWx. 

W2 

The  set  of  all  possible  METOC  outcomes  in  the  engagement  zone,  affecting  the 
performance  of  Blue  air-to-surface  missiles  (formerly  denoted  W ) 

W2 

The  METOC  outcome  in  the  engagement  zone  at  the  time  of  the  missile  exchange 
(formerly  denoted  tv),  w2  eW2.  The  outcome  tv,  determines  which  types  of  Blue 

and  Red  missiles  will  be  usable  during  the  missile  exchange,  as  indicated  in  Table  3, 
page  18. 

v(ax,wx,a2,w2) 

The  mission-relevant  consequences  of  decisions  and  METOC  events,  measured  as  the 
number  of  Red  landing  vessels  that  reach  the  island. 

% 

The  set  of  all  possible  climatologies  that  could  be  issued  in  any  year,  relevant  to  the 
potential  bomber  stations,  in  this  case  modeled  as  ={  El  Nino,  La  Nina,  normal} 

Pi 

The  climatology  issued  before  the  bomber  stationing  decision  is  made, 

The  set  of  all  states  of  information  possible  METOC  forecasts  available  for  the 
engagement  zone,  formerly  denoted  as  ®  in  Section  3.4,  ®2  ={1,2, 3. 4}. 

<P2 

A  single  possible  forecast  for  METOC  conditions  in  the  engagement  zone  at  the  time 
of  the  mission  exchange,  (p2  e  ®2 .  Can  be  interpreted  as  a  categorical  forecast  of  the 

METOC  outcome  tv, . 
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5.2  Outcomes 

There  are  now  two  scenario-relevant  METOC  outcomes. 

5.2.1  METOC  conditions  in  the  engagement  zone 

The  METOC  outcome  wsW  described  in  Section  3.2  now  becomes  w2  e  W2  and  determines  whether 
missiles  of  Type  A  and  B  can  be  used  in  the  engagement  zone  as  indicated  in  Table  3  on  page  18.  In 
addition,  when  vv,  =  4,  the  length  of  Red’s  transit  is  prolonged.  This  affects  mine-laying  but  does  not 

affect  the  outcome  of  the  missile  exchange.  With  the  parameter  values  used,  the  length  of  the  engagement 
does  not  affect  the  number  of  Red  vessels  surviving  the  exchange,  because  either  Red  vessels  or  Blue 
aircraft  are  eliminated  within  six  salvos,  with  the  parameters  used  here  (See  Appendix  A). 

5.2.2  METOC  conditions  at  bomber  bases 

An  additional  METOC-outcome  variable  has  been  introduced,  which  reflects  METOC  conditions  at  the 
bomber  station.  The  variable  wleWl=  {1,2, 3, 4}  represents  this  METOC  outcome  and  is  defined  in 

Equation  (4)  so  that  it  indicates  simply  whether  conditions  allow  bombers  to  fly  out  of  each  base  -  details 
as  to  why  conditions  allow  or  disallow  the  mission  are  not  modeled. 

Definition 

1  All  Bases  OK,  18-hour  Red  transit 

(4) 

2  Kadena  No  Fly, ,  1 8-hour  Red  transit 

W  = 

3  Guam  No  Fly, ,  18-hour  Red  transit 

4  Kadena,  Guam  No  Fly,  36-hour  Red  transit 


Although  the  choice  as  to  where  bombers  will  be  stationed  (al )  is  made  before  the  outcome  w,  is  known, 
the  definitions  of  and  W]  do  not  depend  on  ai ,  and  reflect  conditions  at  all  of  the  possible  bomber 
stations.  Therefore,  if  bombers  are  stationed  at  Guam,  =1  and  vv,  =  2  have  identical  effects  on 
consequences,  as  do  =  3  and  vv,  =  4.  Note  that,  according  to  the  model,  METOC  conditions  at  the 
Alaska  base  will  always  permit  flying  -  there  is  no  chance  that  bombers  stationed  in  Alaska  may  be 
prevented  from  operating  due  to  METOC  conditions. 

5.3  Consequences 

In  the  Halsey  scenario,  the  consequence  measure  has  been  selected  as  the  number  of  Red  landing  vessels 
surviving  to  land.  This  determines  the  size  of  the  amphibious  landing  force  that  reaches  the  defended 
island,  and  therefore  reflects  the  most  mission-critical  consequence  of  the  mine  and  missile  exchange. 
Again,  as  discussed  in  Section  3.3,  the  costs  of  sorties  and  missiles,  as  well  as  mines  and  mine-removal, 
are  treated  as  negligible.  If  these  consequences  are  significant  relative  to  number  of  Red  landing  vessels, 
then  they  should  be  quantified  and  brought  in  using  a  multi-attribute  measure  of  consequence. 

Under  the  extended  scenario,  two  stages  of  Blue  decisions  and  METOC  outcomes  determine  the 
consequences.  Therefore,  the  notation  must  be  expanded  to: 
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v(ax,  w, ,  a2,  w2 )  =  number  of  Red  escort  vessels  landing. 

The  consequences  also  depend  on  the  Red  missile-selection  decision.  However,  it  is  assumed  that  Red  is 
able  to  choose  its  surface-to-air  missile  type  with  perfect  information  regarding  the  METOC  conditions  in 
the  engagement  zone,  and  will  choose  the  missile  type  that  is  most  effective  under  the  prevailing 
conditions.  Therefore,  the  METOC  outcome,  w2 ,  uniquely  determines  the  Red  missile  selection,  and  we 

suppress  notation  indicating  the  dependence  of  v(-)  on  Red’s  decision. 

The  task  at  hand  is  to  model  the  function  ,  which  can  be  represented  with  a  3x4x2><4 

matrix. 16  The  modeling  of  the  missile  exchange  is  unchanged,  and  is  described  in  Section  3.3. 

5.3.1  Mining  Model 

The  results  of  the  missile  exchange  determine  the  number  of  Red  escort  and  landing  vessels  that  enter  the 
minefield.  Depending  how  many  mines  have  been  laid,  some  fraction  of  the  Red  landing  vessels  can  be 
expected  to  be  disabled  by  mines.  The  remaining  landing  vessels  survive  to  land. 

It  is  assumed  that  the  mines  are  laid  very  close  to  the  landing  beach,  and  therefore,  the  Red  force  crosses 
the  mine  field  after  completion  of  the  missile  exchange  -  in  other  words.  Blue  aircraft  do  not  continue  to 
strike  the  Red  vessels  when  they  are  very  close  to  shore,  in  the  surf  zone. 

The  fraction  of  Red  vessels  disabled  by  the  minefield  depends  on  how  many  mines  are  laid.  It  is  assumed 
that  the  density  of  mines  and  the  size  of  the  transiting  vessel  determine  the  likelihood  that  a  given  vessel 
will  be  disabled  in  transit.  Specific  assumptions  regarding  the  triggering  radius  and  effective  radius  of  the 
mines,  the  probability  that  a  single  mine  will  be  sufficient  to  disable  a  vessel  are  given  in  Appendix  C. 

The  density  of  mines  is  determined  by  the  number  of  mines  that  the  bomber  fleet  can  drop  -  which 
depends  on  how  far  the  fleet  is  stationed  from  the  engagement  zone,  as  well  as  on  characteristics  of  the 
landing  beach  and  mined  area.  Assumptions  about  the  landing  beach  and  mined  area,  as  well  as  the 
formula  for  fraction  of  Red  vessels  destroyed  are  given  in  Appendix  C. 

The  number  of  mines  the  bomber  fleet  can  drop  depends  on  how  many  sorties  they  can  complete  before 
the  Red  force  has  transited  the  engagement  zone.  The  closer  the  bombers  are  to  the  engagement  zone,  the 
more  round-trip  sorties  they  can  complete.  If  the  bombers  are  stationed  at  Kadena,  they  can  complete  a 
sortie  and  be  ready  to  go  again  in  3  hours,  whereas  if  they  are  stationed  at  Guam,  the  sortie  takes  4  hours, 
and  from  Alaska,  it  takes  6  hours.  If  METOC  conditions  in  the  engagement  zone  are  relatively  good 
( w2  =1,2  or  3),  the  Red  fleet’s  transit  takes  18  hours,  meaning  that  bombers  stationed  at  Kadena  could 
complete  2  sorties,  while  bombers  at  Alaska  can  complete  only  three  before  the  Red  fleet  reaches  the  surf 
zone.  If  METOC  conditions  are  poor  (w2=  4),  the  Red  fleet’s  transit  time  is  36  hours,  which  allows 
proportionately  more  sorties  from  each  station. 

Table  9  shows  the  results  of  the  mining  model,  in  terms  of  the  fraction  of  Red  vessels  disabled  as  a 
function  of  ax ,  and  w2  .  As  expected,  stationing  at  Kadena  (  al  =1)  or  Guam  ( al  =2)  is  more  desirable 
when  METOC  conditions  permit  flying  out  of  these  bases.  Alaska  is  a  more  desirable  location  when  the 
transit  is  long  ( w2  =4)  than  when  the  transit  is  short  ( vv2  =1,2,3). 

Table  9:  Fraction  of  Red  vessels  disabled  by  minefield,  as  a  function  of  bomber  stationing  (Ufj)  and  METOC 

outcomes  ( wl  and  w2 ). 


16  There  are  three  possible  values  of  al ,  four  possible  values  of  ,  two  possible  values  of  a, ,  and  four  possible 
values  of  w2 . 
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a\ 

w1  =1 

W2  =1,2,3 

Wj  =2  Wj  =3 

Wj  =4 

Wj  =1 

w2 

Wj  =2 

= 4 

wl  =3 

Wj  =4 

1 

0.40 

0.00 

0.40 

0.00 

0.76 

0.00 

0.76 

0.00 

2 

0.31 

0.31 

0.00 

0.00 

0.61 

0.61 

0.00 

0.00 

3 

0.15 

0.15 

0.15 

0.15 

0.40 

0.40 

0.40 

0.40 

5.4  Information  Sets 

The  METOC  forecast  and  event  described  in  Section  3.4  will  still  be  used,  but  given  that  an  additional 
information  set  (climatology)  and  an  additional  METOC  event  (describing  the  METOC  conditions  at 
bomber  stations)  have  been  added,  the  former  forecast  (p  and  METOC  outcome  w  are  now  denoted  (p2 

and  w2 .  The  addition  of  climatology  does  not  change  the  probabilistic  relationship  between  (p2  and  w2  . 
Therefore,  P\w2  \  (p2  ]  are  given  in  Table  5. 

The  addition  of  METOC  event  wl ,  describing  the  METOC  conditions  prevailing  at  bomber  stations  at  the 
time  that  Red  initiates  hostilities  was  described  in  Section  0.  An  additional  information  set  ( cpt )  indicates 

which  of  three  modes  -  El  Nino,  La  Nina,  or  neither  -is  expected  to  prevail.  The  climatology  would 
include  information  about  METOC  conditions  typically  associated  with  each  mode  -  minimum,  mean  (or 
mode)  and  maximum  values  for  key  parameters.  This  information  does  not  have  to  be  modeled  to 
determine  decisions,  consequences,  and  therefore  climatology  value.  Rather,  the  probabilistic  relationship 
between  the  possible  climatology  realizations  and  METOC  outcomes  is  all  that  is  needed  to  determine 
decisions.  If  there  were  a  greater  range  of  possible  climatology  realizations  -  if,  for  example,  a 
climatology  could  indicate  mild  vs.  severe  El  Nino,  or  El  Nino  with  high  SST  but  low  wind  shear,  then 
the  set  Oj  would  have  to  be  expanded  to  include  all  possibilities.  At  the  same  time,  the  conditional 

probabilities  of  all  values  of  wx<eWx,  conditional  on  all  possible  would  need  to  be  estimated.  We 

model  P\wx  |  <^,]  and  P\<P\\  as  given  in  Table  10  below. 

In  addition,  it  is  assumed  that  climatology  conditions  the  forecast  for  the  engagement  zone.  This  indicates 
that  certain  METOC  conditions  are  more  (less)  likely  under  each  ENSO  mode  and  therefore  forecasts  of 

certain  METOC  conditions  are  more  (less)  likely  given  the  climatology  issued.  Therefore  P[<p2  <7>i  ]  is 
modeled,  as  shown  in  Table  10.  The  probability  distribution  is  relatively  spread  out  -  i.e.  the  highest 
valued  P\(p2  \  (p{\  is  0.45,  indicating  that  relatively  little  uncertainty  about  the  eventual  METOC 

outcomes  in  the  engagement  zone  can  be  predicted  at  the  time  of  the  climatology;  perhaps  a  more  realistic 
model  would  make  the  climatology  even  less  informative. 
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Table  10:  Probability  distributions  of  climatology,  METOC  outcomes  at  bomber  bases,  and  relationship 
between  climatology  and  forecasts. 


p[wx\(px] 

1 

Wi 

2 

3 

4 

PM 

1 

0.00 

0.30 

0.10 

0.60 

0.33 

2 

0.65 

0.25 

0.10 

0.00 

0.33 

3 

0.33 

0.28 

0.10 

0.30 

0.33 

P\(P2\(P\] 

1 

(p2 

2 

3 

4 

1 

0.10 

0.15 

0.30 

0.45 

2 

0.45 

0.30 

0.15 

0.10 

3 

0.25 

0.25 

0.25 

0.25 

<Pi 


<Px 


5.5  Decision  Policies 


As  described  in  Section  3.5,  the  decision  policies  are  determined  by  optimizing  expected  total 
consequences  in  each  possible  decision  situation  that  might  arise.  Because  more  than  one  forecast  (or 
climatology)  might  be  issued,  there  are  multiple  decision  situations.  A  decision  policy  describes  the  action 
that  would  be  selected  in  every  possible  situation  -  for  example,  for  each  possible  METOC  forecast. 

As  described  in  Section  4.4,  In  a  multi-stage  decision  context,  the  number  of  decision  situations 
multiplies.  For  each  forecast  cp , ,  previous  events  -  including  the  selected  value  of  ax ,  the  climatology  <px 

and  the  METOC  outcome  wx  -  may  all  affect  the  optimal  decision  for  a  given  value  of  (p2 ,  hence: 17 


a2  {(px,wx,ax,(pj)  =  min  E 

#2 


VP2|^2  >fll  >w\ 


v(ax,wx,a2,w2)]. 


Earlier  decisions  (in  this  example,  ax )  are  made  based  only  on  information  available  and/or  decisions 

made  prior  to  that  time.  Therefore,  in  the  Halsey  scenario,  ax  =  ax  (<^ ) .  As  discussed  in  Section  4.4,  we 

cannot  generally  write  the  solution  to  the  early  decisions  in  the  form  of  minimizing  the  expectation  of  a 
single  expression  -  rather,  we  must  solve  for  these  using  backward  induction.  Because  of  the  simple 
structure  of  the  Halsey  scenario  modeled,  the  fraction  of  ships  disabled  by  the  minefield  is  independent  of 

a2  and  vv2  and  depends  only  on  ax  and  wx .  Therefore  a‘  ((p} )  simply  maximizes  the  expected  fraction 
of  ships  entering  the  minefield  that  are  disabled,  where  expectation  is  taken  over  values  of  wx , 
conditioned  on  <px . 


The  decision  process  is  illustrated  in 
A  if  (p2  =  1 
A  if  (p2  =  2 
B  if  <p^  =  3 
[B  if  cp2  =  4 


a2{cpx,wx,ax,(p2)  =  \ 


Appendix  B.  In  this  case,  it  turns  out  that  the  optimal  decisions 


V  (px ,  wx  and  ax . 


17  The  specific  structure  of  the  Halsey  scenario  would  allow  us  to  suppress  Wx  and  <7  in  defining  the  model,  but  the 
solution  procedure  requires  preserving  the  dependence  of  v(-)  and  U-,  (•  )  on  Wx  and  <7, . 
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The  optimal  decisions  at  the  first  opportunity,  a]  {(p] )  = 


Alaska  if  tpx  =  1 
Kadena  if  q\  =  2  . 
Guam  i  1'  cp]  =  3 


The  ex  ante  (before  receipt  of  climatology  (px )  expected  number  of  Red  vessels  landing  is  48.8.  It  is 
interesting  to  note  that  the  climatology  associated  with  the  best  outcomes  (from  Blue’s  perspective)  is 
(px=\,  which  predicts  the  highest  probability  of  severe  METOC  in  both  the  engagement  zone  and  the 

bomber  bases.  Because  in  this  model  Blue’s  missiles  perform  in  more  METOC  conditions  than  Red’s,  it 
is  to  Blue’s  advantage  to  have  the  engagement  occur  in  severe  METOC  conditions.  If  Red’s  decision 
model  were  more  sophisticated,  we  might  include  Red’s  ability  to  time  their  commencement  of  hostilities 
to  coincide  with  favorable  METOC  conditions. 

Table  11:  Expected  total  consequences  for  bomber  stationing  decision,  given  optimal  missile-selection 
decision  on  the  basis  of  later  forecasts. 


<Pl 

v(al,w1,a2  =  a2(fw2) 

1 

a\ 

2 

3 

1 

48.6 

45.7 

42.5 

(Pi 

2 

52.9 

54.0 

64.6 

3 

52.3 

50.9 

53.0 

As  Table  1 1  indicates,  the  difference  between  the  best  and  worst  decisions  is  greatest  for  cpx  =2  (a 
difference  of  1 1.7  Red  vessels,  in  expectation,  as  compared  with  a  difference  of  only  2.1  vessels  when 
(p]  =3).  As  will  be  illustrated  and  discussed  in  the  following  sections,  the  VOI  may  be  higher  when  the 

variability  in  outcomes  across  decisions  and  METOC  conditions  is  greatest,  not  necessarily  when  the 
consequences  are  of  highest  magnitude. 


5. 6  Value  of  Information 

As  discussed  in  Section  4.5,  estimating  the  VOI  requires  modeling  a  less-informative,  reference  case, 
identifying  what  the  commander  would  do,  and  what  consequences  would  ensue,  if  she  did  not  have 
information  available.  Since  we  are  estimating  the  value  of  climatology  information  in  particular,  we  will 
consider  what  the  commander  would  do  with  neither  climatology  nor  a  forecast  (Section  5.6.1)  and  with 
no  climatology,  but  with  a  later  forecast  (Section  5.6.2). 


5.6.1  Comparison  with  no  information 

If  the  commander  neither  received  nor  anticipated  any  forecast  information  she  would  make  both  the 
bomber  stationing  decision  and  the  missile-selection  decision  on  the  basis  of  the  long-run  average 
occurrence  of  the  METOC  conditions  affecting  the  consequences  of  each  decision.  She  would  not  need  to 
anticipate  multiple  possible  realizations  of  the  forecast  information  set,  but  would  simply  look  at  the  long- 

run  average  rate  of  occurrence  of  each  w2  e  W2  and  each  w  ,  e  Wx  and  choose  the  pair  (a1?a,)  that 

produces  the  best  (lowest)  expected  consequences.  Therefore,  her  decision  rule  is: 


min 

(a[  ,ch  )e(5  >T) 


wva2,w2 


)] 


(3) 
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where  the  expectation  will  be  taken  with  respect  to  the  -  unconditioned  -  joint  probability  distribution  of 
w  j  and  w2 ,  calculated  from  Table  5  and  Table  10, 18  and  the  consequences  given  in  Table  12  below. 

Table  12:  Expected  consequences  of  each  pair  of  stage  1  and  stage  2  alternatives  (ax  and  a2 )  and  METOC 
outcomes  using  unconditioned  (marginal)  probability  distribution  of  METOC  outcomes,  W1  and  W2. 

a\ 


The  optimal  decisions,  a  (^j)and  a  [al,(pRX^  are  a  (^fll)=G  (Guam)  and  a  (a*x,(pR^  =  B,  which 
yield  an  expected  60.2  Red  vessels  landing.  The  VOI  is  1 1.4  vessels  =  60.2  -  48.8. 19 

5.6.2  Comparison  with  forecast  only  -  no  climatology 

If  the  commander  had  forecasts  available,  but  not  climatology,  her  best  policy  would  be 


A  if  cp2  =  1 
* ,  \  A  if  (p2  =  2 

a2  ( (px ,  w1 ,  al ,  (p2 )  =  \  ~  ,  V  (px ,  Wj ,  ax ,  and  the  best  policy  for  choosing  a  x  would  be 

B  if  (p2  =  3 

B  if  cp2  =  4 


la1,w1,a2 1 


)  |  =  G  (Guam),  for  an  expected  consequence 


of  50.2  Red  vessels  making  landing.  The  VOI  for  forecasts  is  therefore  60.2  -  50.2  =  10,  and  the  VOI  of 
adding  climatology  to  the  forecast  is  50.2  -  48.8  =  1.4. 


In  light  of  Blue’s  objective  to  reduce  the  expected  number  of  Red  vessels  making  landing,  we  should 
anticipate  that  the  expected  consequence  of  having  forecasts  but  no  climatology  would  be  greater  than  in 
the  case  with  both  climatology  and  forecasts,  but  less  than  the  case  with  neither.  As  summarized  in  Table 
1 3  below,  this  is  what  we  find. 


18  P[wvwi\  =  X  (PM-p[wi\<PilP[w2\<Pi])  where  p[wi \<Pi]=  P^Z2  ^  » 

P{(Px,W2 ]=  X  \<p2\P{(Px  l^2])>  P[<Pi  I  (Pi\=  P f  .  P[(Pi]  and 

<f>2  g®2  PVPl\ 

P [  VV|  |  </?!  j  are  given  in  Table  10,  P  [  VV2  |  (p2  j  and  P  [ (p2  ]  are  given  in  Table  5.  The  above  formulas  assume  that 

P[w,  I  (pt  ]  and  p[w2  (p]  ]  are  independent,  and  P [  W2  (p2  ]  and  P [ (p]  (p2  ]  are  independent. 

19  The  VOI  in  the  two-stage  Halsey  scenario  (11)  is  close  to  the  VOI  calculated  for  the  single-stage  scenario  (12). 
This  similarity  is  coincidental  -  the  two-stage  scenario  is  quite  different,  and  adds  the  possibility  of  mining  the  surf 
zone.  The  expected  number  of  Red  vessels  making  landing  is  thereby  reduced,  and  there  is  no  clear  relationship 
between  the  VOI  in  the  two  scenarios. 

20  The  fact  that  OC2  does  not  depend  on  (px ,  Wj  and  ax  is  an  artifact  of  this  scenario;  in  general,  there  could  be  a 
dependence. 
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Table  13:  Value  of  Information  and  expected  consequences  using  no  information,  forecasts  only,  and 
forecasts  with  climatology. 


A[v(fl1,rv1,a2,tv2)] 

Expected  Red 
vessels  landing 

VOI 

relative  to  no 
information 

VOI 

relative  to 
forecast 
only 

VOI 

relative  to 
climatology 
only 

Both  forecasts  and  climatology 

48.8 

11.4 

1.4 

9.9 

Forecast  only 

50.2 

10.0 

— 

— 

Climatology  only 

58.7 

1.6 

— 

— 

Neither  forecast  nor  climatology 

60.2 

— 

— 

— 

The  value  contributed  by  climatology  (1.4  Red  vessels)  over  and  above  the  value  of  METOC  forecasts  is 
small,  relative  to  the  value  contributed  by  forecasts  (10  Red  vessels).  This  result  is  specific  to  the  Halsey 
decision  and  engagement  model  used  here,  and  may  not  represent  the  general  relative  importance  of 
METOC  climatology  and  forecasts.  However,  we  should  expect  the  incremental  value  of  climatology 
used  in  conjunction  with  forecasts  to  be  lower  than  the  value  of  forecasts  themselves.  Forecasts  are  more 
accurate,  and  therefore  provide  more  informational  value.  Moreover,  the  value  of  climatology  is  properly 
calculated  as  an  addition  to  the  value  of  forecasts  -  if  forecasts  were  not  available,  the  value  of 
climatology  would  (usually,  but  not  necessarily)  be  substantially  higher  than  its  value  in  conjunction  with 
forecasts.  In  addition,  it  should  be  considered  that  this  is  in  expectation,  and  in  certain  scenarios  will  be 
much  higher. 

5.6.3  Comparison  with  climatology  only  -  no  forecast 

The  value  of  adding  climatology  only  to  the  no-information  scenario  may  also  be  calculated.  This 
describes  a  scenario  in  which  the  commander  does  not  receive  any  forecast  for  METOC  conditions  in  the 

engagement  zone  and  therefore  use  long-run  average  frequencies  of  the  METOC  outcomes  (  P[u2  \<Pi]) 
to  make  the  missile-selection  decision  ( a, ). 

The  commander  would  not  need  to  anticipate  multiple  possible  realizations  of  the  forecast  information 
set,  but  would  simply  look  at  the  long-run  average  rate  of  occurrence  of  each  pair  (  ft ,  ft  )  given  ft ,  and 

choose  the  pair  (a1?a2)  that  produces  the  best  (lowest)  expected  consequences.  Therefore,  her  decision 
rule  is: 

(«*,«2*)  =  ,  min  E ,  [v(a1;  ft,a2,ft)l ,  a  function  of  ft  .(3) 

\a\  >a2)E{A’-™2) 


The  results  are 


a, 


Alaska  if  ft  =  1 
Kadena  if  ft  =  2 ,  and 
Guam  if  ft  =  3 


2V^e$j,  with  corresponding 


E 


wi,w2m 


,ft,ft 


'48.6  if  ^  =1 
<  66.0  if  ft  =  2  . 
61.4  if  ft  =3 


The  expected  total  consequences  (with  expectation  taken  over  ft )  is  58.6,  meaning  the  VOI  of 
climatology  only  in  the  Halsey  two-stage  scenario  is  1.6.  When  evaluated  conditioned  on  specific  values 
of  ft  ,  however,  the  value  of  climatology  is  higher.  When  ft  = 2 ,  climatology  does  not  affect  a1  and 

therefore  has  no  value,  as  defined  with  the  VOI  framework.  On  the  other  hand,  when  ft  =1  or  3, 
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climatology  will  induce  the  commander  (if  she  behaves  optimally)  to  change  her  bomber  stationing 
decision.  In  the  case  of  (px  =1,  this  increases  the  chances  that  mining  will  be  possible  -  because  bombers 
will  be  flown  out  of  Alaska,  reducing  the  expected  number  of  Red  vessels  landing  from  51.8  to  48.6  (a 
difference  of  3.2).  In  the  case  of  (p]  =3,  stationing  bombers  at  Kadena  rather  than  Guam  increases  the 

number  of  sorties  that  can  be  completed  in  the  (relatively  likely)  event  of  permissible  METOC  conditions 
for  flying,  thus  reducing  the  expected  number  of  Red  vessels  landing  from  64.2  to  61.3,  a  difference  of 
2.8. 

6  Value  Insights 

This  section  describes  and  illustrates  -  using  the  Halsey  scenario  and  other  examples  -  factors  that 
generally  increase  (or  decrease)  the  VOI  contained  in  forecasts,  ending  (Section  6.4)  with  a  discussion  of 
the  VOI  for  early  forecasts  (or  climatology)  when  there  are  also  later  forecasts. 

6.1  How  alternatives  affect  value 

The  type  of  alternatives  available  and  the  way  these  alternatives  mediate  the  impact  of  METOC 
conditions  are  a  major  determinant  of  the  value  of  forecast  information.  As  discussed  earlier,  the  common 
assumption  that  information  is  always  valuable  and  that  more  accurate  forecasts  always  add  value  may 
not  necessarily  hold.  The  value  of  forecasts  depends  what  can  be  done  in  response  to  the  information 
they  contain  -  in  other  words,  forecast  value  depends  on  the  alternative  courses  of  action  available. 

6.1.1  Decision-sensitive  consequences 

Information  provides  greater  value  when  decisions  substantially  influence  outcomes  -  i.e.  for  a  given 
METOC  outcome,  there  is  a  significant  difference  in  consequences  depending  on  which  course  of  action 
is  selected 

Depending  on  the  decision  context,  consequences  may  be  METOC-sensitive  but  not  decision-sensitive  or 
vice  versa.  When  consequences  are  METOC-sensitive  but  not  decision-sensitive,  even  highly 
accurate  METOC  forecasts  of  operationally  relevant  parameters  may  be  of  low  value  -  and  in  some 
cases  should  even  be  ignored  by  rational  decision  makers.  A  classic  example  is  when  severe  METOC 
conditions  are  threatening  to  life  and  health  (i.e.  consequences  are  highly  METOC-sensitive)  -  but  there 
is  little  or  nothing  to  do  about  it.  In  the  Halsey  scenario,  if  commanders  learned  24  hours  before  their 
mission  that  there  is  a  high  probability  of  a  tropical  storm  affecting  Guam  and  Kadena,  it  would  be  too 
late  to  re-station  the  bombers.  At  this  point,  there  is  no  remaining  alternative  but  to  wait  and  see  if 
METOC  conditions  will  permit  flying,  and  there  is  no  value  to  seeking  the  24-hour  forecast  for  the 
bomber  stations. 

A  similar  situation  occurs  when  alternatives  are  available,  but  alternatives  that  would  affect  METOC 
consequences  -  e.g.  protect  life  and  health  are  as  costly  as  the  potential  METOC  damages  themselves. 

For  example,  if  the  consequences  of  failing  to  complete  a  mission  are  so  severe  that  it  is  worthwhile  to 
incur  substantial  risks,  then  METOC  forecasts  may  be  ignored.  Even  if  the  forecaster  is  very  confident 
that  conditions  will  be  adverse,  the  mission  may  proceed  -  and  METOC  information  has  little  or  no  value. 
Rules  that  allow  president-ordered  operations  to  proceed  even  when  operational  thresholds  for  METOC 
and  other  conditions  are  violated  demonstrate  that  sometimes  the  value  of  even  a  small  probability  of 
mission  success  is  enough  to  outweigh  METOC-related  risks.  By  contrast,  for  low-value  missions, 
METOC  information  (climatology  or  forecasts)  may  be  highly  valuable  because  small  differences  in 
METOC  conditions  will  drive  decisions. 
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6.1.2  Flexibility 

Information  provides  greater  value  when  decision  makers  have  flexibility  -  i.e.  more  alternatives 
(generally)  mean  more  value.  We  can  even  make  the  stronger  statement  that,  if  the  decision  maker 
behaves  normatively,  the  addition  of  an  alternative  cannot  lower  VOI,  all  else  remaining  unchanged. 

A  simple  intuitive  explanation  of  this  relationship  is  that  the  more  alternatives  are  available,  the  more  the 
selected  alternative  will  be  fine-tuned  to  the  METOC  forecast.  At  the  limit,  if  there  are  no  alternatives  - 
i.e.  the  decision-maker  has  no  decisions  to  make  -  then  information  has  no  value,  as  discussed  earlier. 

Factors  that  contribute  to  flexibility  -  and  therefore  to  higher  value  for  forecasts  -  include: 

•  Mobile  assets  (geographic  flexibility)  -  when  forecasts  are  good,  mobile  assets  can  be  moved  to 
follow  desirable  METOC  conditions.  Commonly,  this  is  operationalized  when  valuable  assets  are 
moved  to  avoid  adverse  conditions  that  might  damage  them  -  for  example,  aircraft  are  moved 
when  severe  METOC  is  expected,  and  ships  are  routed  to  avoid  high  seas.  Value  arises  from  the 
prevention  of  damage. 

More  generally,  when  it  is  possible  to  conduct  METOC-sensitive  operations  (or  store  METOC- 
sensitive  assets)  in  multiple  locations,  METOC  forecasts  are  highly  valuable  in  identifying 
locations  with  desirable  conditions. 

•  Timing  flexibility  -  the  ability  to  move  up  or  delay  an  operation  means  that  METOC-sensitive 
activities  can  be  conducted  during  times  when  METOC  conditions  are  more  suitable  (relative  to 
what  they  might  be  if  timing  were  fixed). 

This  has  the  counterintuitive  result  that  METOC  forecasts  may  be  more  valuable  for  routine 
missions  than  for  more  important  missions.  Less-critical  missions,  such  as  training,  generally 
have  more  timing  flexibility  than  critical  missions.  If  the  mission  is  extremely  urgent  (as  in 
search-and-rescue)  or  has  a  very  narrow  time  window,  then  the  mission  will  go  ahead  regardless 
of  METOC  forecast  and  METOC  conditions. 

A  few  hours’  wiggle  room  may  be  enough  to  exploit  the  value  of  METOC  forecasts  -  in  the 
planning  for  D-Day  in  1944,  the  flexibility  across  just  a  few  days  allowed  the  Allies  to  exploit  a 
small  window  in  METOC  conditions. 

•  Short  response  times  -  if  there  are  multiple  alternatives  that  can  be  selected  with  short  lead 
times  before  the  relevant  METOC  events,  more  flexibility  is  preserved  until  late  in  the 
operational  planning.  As  forecast  accuracy  generally  improves  as  lead  time  declines,  short 
response  times  allow  for  decisions  to  be  made  with  better  information  -  resulting  in  (on  average) 
better  outcomes.  For  example,  if  a  decision  must  be  made  with  a  24-hour  lead  time,  a  12-hour 
forecast  may  not  have  value. 

A  related  concept  -  (ir)reversibility  -  is  important  when  there  are  multiple  forecasts  for  a  single 
event  (e.g.  when  climatology  is  available,  followed  by  a  forecast  or  forecasts).  Reversibility  has 
an  ambiguous  relationship  with  value,  as  discussed  in  Section  6.4. 

6.2  METOC  sensitivity 

The  range  of  METOC  outcomes  and  the  relationship  between  consequences  and  METOC  outcomes  are 
related  to  the  VOI  -  in  both  expected  and  unexpected  ways. 

6.2.1  METOC-sensitive  consequences 

Forecasts  are  more  valuable  when  consequences  are  substantially  different  as  a  function  of  the  METOC 
outcome.  In  other  words,  if  METOC  conditions  have  a  big  impact,  it  is  (generally)  more  important  to 
have  accurate  information  about  METOC  conditions  -  all  other  things  being  equal.  An  important  qualifier 
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is  that  there  have  to  be  alternatives,  available  after  the  receipt  of  a  forecast,  to  alter  the  METOC 
consequences  (see  Section  6.1). 

6.2.2  Variability  in  METOC  conditions 

A  corollary  to  the  previous  insight  is  that  a  high  degree  of  variability  in  METOC  outcomes  tends  to  raise 
the  value  of  information.  Variability  in  this  case  means  a  wider  prior  distribution  of  outcomes,  which  is 
distinct  from  predictability. 

An  intuitive  explanation  is  that,  when  METOC  conditions  are  highly  variable,  conditions  suitable  for  each 
alternative  are  relatively  likely  to  arise  -  by  contrast,  when  METOC  conditions  are  very  stable,  the 
alternative  most  appropriate  for  the  common  conditions  will  usually  be  appropriate,  and  only  rarely  would 
forecasts  indicate  that  this  should  change. 

However,  this  relationship  is  even  stronger  than  the  preceding  explanation  indicates.  When  METOC 
conditions  are  more  variable,  rational  decision-makers  should  give  more  weight  to  forecasts  than 
when  METOC  conditions  are  more  stable.  In  normative  decision-making,  the  choice  of  an  alternative 
should  be  driven  by  the  probability  distribution  of  METOC  outcomes  conditional  on  the  forecast  message 
received.  When  the  climatological  variability  is  low,  the  probability  of  extreme  events  will  be  lower, 
conditioned  on  each  possible  forecast.21  A  related  effect  is  elaborated  in  the  following  section. 

The  relationship  between  variability  and  value  is,  however,  context  dependent.  In  considering  submarine 
detection  problems,  for  example,  it  may  be  that  it  is  better  to  search  in  a  region  that  has  a  high  degree  of 
variability  in  detection  conditions  than  in  a  region  with  better  average  conditions,  but  lower  variability.  In 
a  search  problem,  for  example,  multiple  opportunities  for  detection  occur,  and  if  the  enemy  submarine 
passes  through  a  high-detectability  region  once,  this  may  be  enough  to  allow  its  detection.  High- 
detectability  may  be  more  common  in  a  high-variability  region  with  worse  average  conditions  than  in  a 
low- variability  region  with  better  average  conditions. 

6.2.3  Base  rate  of  adverse  METOC  events 

When  the  base  rate,  or  long-run  average  (climatological)  frequency  of  a  critical  METOC  outcome  is  very 
low,  then  the  value  of  a  METOC  information  system  is  low.  This  is  caused  by  two  factors: 

1 .  If  the  critical  outcome  occurs  infrequently,  opportunities  for  METOC  information  to  influence 
decision-makers  to  prepare  for  the  critical  outcome  are  rare,  and  therefore,  the  long-run  average 
value  contributed  by  the  METOC  information  will  below;  and 

2.  If  the  base  rate  for  the  critical  outcome  is  low,  then  the  probability  of  the  critical  outcome 
occurring  -  even  conditional  on  the  issuance  of  a  forecast  of  the  critical  outcome  -  is  relatively 
low.  This  would  hold  even  for  highly  accurate  forecasting  systems.  Therefore,  a  decision-maker 
might  rationally  choose  to  plan  for  other,  more  likely,  outcomes.  From  the  forecaster’s 
perspective,  this  could  be  inteipreted  as  the  decision-maker  ignoring  the  forecast.  The  value  of  the 
forecast  would  therefore  be  low. 

Like  the  other  value  insights,  the  effect  of  base  rate  applies  when  all  else  is  held  constant.  In  a  situation 
where  the  decision  is  very  sensitive  to  the  METOC  conditions,  or  the  ability  of  decisions  to  affect  the 
impact  of  the  critical  METOC  outcome  is  very  high,  then  forecast  information  may  be  highly  valuable, 
even  if  the  base  rate  for  the  critical  METOC  outcome  is  low.  Tropical  cyclone  forecasting  is  a  good 
example  -  although  the  frequency  of  tropical  cyclone  conditions  at  any  given  location  is  relatively  low, 
the  consequences  of  failing  to  prepare  for  the  storm  are  so  severe,  that  decision-makers  will  want  to  act  on 
forecasts  even  though  the  base  rate  -  and  therefore  the  probability  of  critical  METOC  conditions 
conditional  on  the  issuance  of  a  forecast  of  tropical  cyclone  conditions  -  is  relatively  low. 


21  This  basic  relationship  implies  that  forecasters  should  regress  to  the  mean  when  METOC  conditions  are  relatively 
stable.  Human  forecasters  rarely  do. 
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6.3  Forecast  Sensitivity 

6.3.1  Accuracy  and  Value 

Generally,  greater  accuracy  leads  to  greater  value  -  because  it  is  less  likely  (less  frequent)  that  the 
forecast  errs,  leading  decision-maker  to  select  an  alternative  that  is  not  appropriate  for  the  actual  METOC 
conditions.22 

However,  this  general  result  is  subject  to  the  effects  of  METOC-sensitivity  and  alternative-sensitivity  of 
the  consequences.  There  may  be  ranges  over  which  increases  in  accuracy  do  not  affect  value,  as,  for 
example,  when  there  are  METOC  conditions  over  which  a  single  alternative  is  best  -  forecasts  that  are 
highly  accurate  at  discriminating  among  these  outcomes  are  valueless. 

6.3.2  Decisions  are  highly  sensitive  to  forecast  in  the  range  of  likely  forecasts 

Information  provides  greater  value  when  the  range  of  possible  forecasts  includes  information  sets  that 
would  induce  substantially  different  courses  of  action.  In  other  words,  all  other  things  being  equal  (even  if 
the  consequences  of  a  bad  METOC  outcome  or  a  retrospectively  bad  decision  are  severe),  if  there  is  a 
high  prior  likelihood  that  the  forecast  will  induce  a  given  decision,  then  there  is  less  value  to  the  forecast. 

If,  on  the  other  hand,  likely  forecasts  would  tip  the  decision  sometimes  one  way  and  sometimes  the  other, 
then  the  forecast  has  greater  value.  This  can  occur,  for  example,  if  forecasts  frequently  (or  likely)  occur 
on  each  side  of  a  threshold. 

To  see  how  this  works,  consider  a  simple  costdoss  decision  model,  under  four  decision  scenarios, 
depicted  in 


22  We  could  even  say  that  if  decisions  are  made  rationally  (optimally),  increasing  accuracy  can  never  decrease  VOI, 
unless  increasing  accuracy  leads  to  less  informative  forecasts  -  e.g.  if  we  are  starting  with  a  forecast  that’s  always 
wrong  (and  therefore  highly  informative!). 
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Table  14  below.  There  are  two  critical  thresholds  -  these  could  reflect  two  different  types  of  mission.  For 
example,  icing  may  occur  if  ground  temperatures  are  <0°C,  so  flights  are  dangerous;  heat  stroke  is  a  high 
risk  if  temperatures  >20°C,  so  field  training  should  be  cancelled.  There  are  also  two  probability 
distributions  of  the  actual  temperature  -  perhaps  the  first  represents  January  and  the  second  represents 
April.  Assume  a  perfect  forecast  is  available,  and  in  the  absence  of  the  forecast,  the  mission  would  always 
proceed. 

Why  does  this  occur?  Information  helps  decision-makers  to  know  when  to  cancel  the  mission  and  avoid 
incurring  the  loss.  For  the  flying  mission,  this  occurs  when  the  temperature  (and  therefore  the  perfect 
forecast)  will  be  less  than  <0°C.  This  occurs  13.5%  of  the  time  under  the  first  probability  distribution,  and 
only  1.3%  under  the  second  probability  distribution.  In  April,  ignoring  the  forecast  means  the  probability 
of  icing  conditions  is  only  1.3%  -  a  much  smaller  risk.  Similarly,  for  the  field  training  mission,  the 
probability  of  exceeding  the  threshold  >20°C  is  less  than  0.1%  for  the  January  probability  distribution, 
and  21.5%  for  the  April  probability  distribution.  In  January,  the  training  commander  can  afford  to  ignore 
the  forecast,  but  in  April,  ignoring  the  forecast  would  mean  running  a  substantial  risk. 
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Table  14:  Effect  of  interaction  between  operational  value  and  probability  distribution  over  METOC 


It  is  not  simply  the  threshold  or  the  base  rates  that  determine  the  value  of  the  forecast,  but  the  intersection 
between  the  two. 


6.4  Climatology 

Information  provides  greater  value  when  decisions  are  difficult  or  impossible  to  undo.  Because 
climatology  information  is  available  at  long  lead-times,  there  may  be  a  wide  scope  for  operational 
decision-making  at  climatological  lead  times.  For  example,  climatology  may  allow  commanders  to  move 
the  timing  of  an  operation  up  or  back  by  weeks  or  months,  rather  than  days  or  hours,  and  to  move  assets 
into  or  out  of  a  theater.  On  the  other  hand,  these  decisions  and  their  implementation  may  be  reversible, 
possibly  at  some  cost.  The  ability  to  reverse  or  undo  previous  actions  (see  flexibility  in  Section  6.1.2) 
generally  declines  (or  cost  of  reversing  increases)  as  lead  time  declines. 

The  key  to  VOI  is  the  relationship  between  receipt  of  information  and  loss  of  reversibility  (flexibility). 

As  discussed  and  illustrated  in  Section  5.6,  the  VOI  for  climatology  depends  on  the  METOC  forecasts 
that  will  be  received  later,  and  how  much  flexibility  will  remain  once  METOC  forecasts  are  received.  The 
more  flexibility  that  remains  later,  the  less  value  climatology  will  contribute. 

Long  lead-time  decisions  that  may  be  informed  by  climatology  generally  create  the  constraints  for  short 
lead-time  decisions  that  can  make  use  of  METOC  forecasts.  Long  lead-time  decisions  may  determine  the 
decision  context,  i.e.  what  alternatives  are  available  (example:  where  are  bombers  stationed),  how  much 
impact  those  alternatives  will  have  (e.g.  by  having  reduced  impact  of  bad  outcomes  or  cost  of  protective 
actions).  Long  lead-time  decisions  may  also  influence  short-lead  decisions  by  affecting  the  likelihood  of 
METOC  forecasts  and  outcomes.  The  simplest  example  is  that  climatology  could  influence  planners  to 
plan  an  operation  for  a  given  time  period  or  geographic  location  where  conditions  are  likely  to  be  most 
favorable. 

In  addition,  when  there  are  multiple  forecasts  (or  climatology  plus  at  least  one  forecast)  and  where 
alternatives  include  the  ability  to  collect  data  to  inform  future  forecasts,  then  early  forecast  have  more 
value.  The  early  forecast  enables  decision-makers  to  increase  the  forecast  accuracy  and/or  the  tailoring  of 
the  forecast  for  the  relevant  decision  by  collecting  observations  that  reduce  the  uncertainties  that  are  most 
critical  to  the  decision  at  hand. 
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7  Challenges 

An  OR/DA  approach,  and  the  VOI  tool,  can  be  tremendously  valuable  in  analyzing  METOC  information 
systems  -  including  both  forecast  and  climatology.  As  discussed  in  Section  1,  VOI  can  serve  the  METOC 
enterprise  in  many  ways,  not  least  by  quantifying  the  value  of  METOC  services,  and  by  guiding  the 
design  of  new  or  improved  METOC  information  systems  (to  maximize  their  value).  However,  there  are  a 
number  of  important  challenges  to  effective  use  of  OR/DA  in  this  context. 

7.1  Modeling  and  measuring  the  value  of  METOC  information 

This  report  has  illustrated  the  estimation  of  the  VOI  for  METOC  information  using  with  a  highly  stylized 
example  -  the  Halsey  scenario.  The  use  of  several  simplifying  assumptions  in  the  Halsey  scenario  model 
points  to  some  of  the  challenges  to  using  the  OR/DA  framework  to  model  a  real  operational  context. 

7.1.1  Modeling  METOC 

Modeling  the  probabilistic  relationships  among  multiple  stages  (climatology,  forecasts,  outcomes)  and 
multiple  variables  is  essential  to  a  VOI  analysis.  The  probability  of  every  possible  state  of  information 
and  the  probability  of  every  outcome,  conditional  on  each  state  of  information  must  be  modeled.  The 
simplest  case  a  single-stage  decision  with  one  forecast  and  one  METOC  outcome  variable  -  is  described 
in  Section  2.4. 

In  many  situations,  information  and/or  outcomes  may  be  multi-dimensional.  For  example,  for  flight 
operations,  two  operationally  relevant  variables  are  cross-winds  and  visibility.  Therefore,  both  of  these 
variables  should  be  forecast,  and  the  outcomes  of  both  determine  the  operational  impacts.  In  general,  the 
two  variables  will  not  be  independent  -  e.g.  for  cross-winds  and  visibility,  a  positive  correlation  would  be 
expected.  Therefore  one  of  the  simplest  models  of  the  probability  distribution  of  the  states  of  information 
would  be  a  bivariate  normal  distribution,  which  would  require  estimation  of  five  parameters. 

The  model  also  requires  the  probability  of  every  METOC  outcome,  conditional  on  each  state  of 
information.  Again,  a  relatively  simple  model  is  a  multi-variate  normal  distribution,  which  again  requires 
the  means  of  each  of  the  four  variables,  plus  the  covariance  matrix  (ten  independent  parameters). 

Moreover,  there  may  be  multiple  decision  stages.  It  is  not  realistic  to  model  the  use  of  climatology  in  a 
single-stage  decision.  As  discussed  above,  it  is  hard  to  imagine  a  scenario  in  which  only  climatology 
information  is  used  to  make  operational  decisions,  and  no  decision  flexibility  remains  when  later,  more 
detailed  METOC  forecasts  become  available. 

The  profusion  of  variables  and  parameters  required  to  estimate  the  probability  model  grows  quickly  when 
multiple  stages  are  added,  as  when  climatology  is  involved. 

The  modeling  requirements  will  demand  either  large  historical  data  sets  of  both  forecasts  and 
METOC  outcomes,  or  substantial  simplifying  modeling  assumptions.23  The  data  demands  depend  on 
many  factors,  but  increase  exponentially  in  the  number  of  operationally  relevant  METOC  variables,  the 
number  of  time  periods  involved  in  decision-making  for  a  given  mission  and  the  number  of  METOC 
information  sets  relevant  to  a  given  valid  period.  In  general,  the  preservation  of  historical  forecast  data  is 
not  intended  to  support  OR/DA  modeling  -  and  therefore  too  little  data  (from  the  perspective  of  OR/DA) 
is  preserved.  There  are  several  of  reasons  for  the  lack  of  data:  the  decisions  about  what  to  collect  were  not 
made  with  the  needs  of  OR/DA  analysts  in  mind;  the  amount  of  potentially  useful  data  is  much  greater 
than  existing  storage  capacity;  and  as  forecasting  models  and  techniques  change,  data  arising  from 
obsolete  models/techniques  may  be  considered  obsolete. 


23  Even  if  there  is  a  great  deal  of  data,  the  probability  model  will  still  be  a  model,  not  a  perfect  representation  of  the 
ocean-atmosphere-information  system 
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Expertise  gaps,  as  well  as  data  availability,  contribute  to  making  the  probabilistic  modeling  of  METOC 
information  challenging.  Moreover,  data  and  expertise  are  complementary.  The  fewer  data  are  available, 
the  more  important  multivariate,  time-series  statistical  modeling  becomes.  Few  people  are  familiar  with 
both  the  METOC  models  and  the  statistical  techniques  required. 

Data  and  expertise  limitations  for  OR/DA  are  discussed  at  greater  length  in  Regnier  (2007)  and  Regnier 
(2008b). 

This  report  focuses  on  measuring  the  VOl  of  an  existing  METOC  system.  Elowever,  one  of  the  most 
powerful  applications  of  VOI  analysis  is  in  designing  new  and  improved  climatology  or  forecasting 
systems.  The  basic  approach  is  to 

1 .  model  the  operation  and  consequences;  then 

2.  model  multiple  hypothetical  (design)  METOC  information  systems; 

3.  identify  the  decision  policies  that  would  be  appropriate  for  a  user  with  each  hypothetical  system; 

4.  calculate  the  VOI  for  each  system;  and 

5.  compare  -  the  higher  VOI  indicates  the  more  effective  system. 

Step  2  above  is  to  model  multiple  hypothetical  METOC  information  systems  -  each  has  the  modeling 
requirements  described  earlier  in  this  section.  The  challenge  of  creating  these  models  is  substantial. 

7.1.2  Modeling  consequences  of  METOC  outcomes 

Modeling  the  consequences  of  METOC  outcomes  is  a  further  challenge  -  and  is  even  less  familiar  to  the 
METOC  community.  In  order  to  use  OR/DA  to  assess  information  systems,  it  is  necessary  to  model  all 
available  alternatives  that  may  be  selected  by  a  decision-maker  using  METOC  information.  It  is  also 
necessary  to  model  how  both  METOC  conditions  and  decisions  made  on  the  basis  of  METOC 
information  affect  operational  consequences  v(a,w) ,  depicted  in  Figure  lb.  Both  alternatives  and 

consequences  should  be  modeled  quantitatively  -  this  is  well  outside  the  scope  of  responsibility  of 
METOC  experts. 

In  addition  to  the  barriers  posed  by  the  domain  responsibility,  this  kind  of  modeling  is  difficult  in  large 
part  because  relevant  data  are  not  collected.  One  of  the  reasons  is  that  there  is  no  responsible  party. 
Operators  worry  about  controlling  consequences  as  much  as  they  can,  and  do  not  necessarily  have  a  use 
for  information  about  quantitative  relationships  between  METOC  and  consequences.  They  also  are 
cognizant  of  many  factors  other  than  METOC  influencing  consequences.  The  METOC  community  - 
including  both  researchers  and  forecasters  -  is  not  generally  familiar  with  operations  and  consequences, 
and  do  not  generally  think  of  the  relationship  between  METOC  and  consequences  as  part  of  their 
responsibility. 

Even  if  the  responsibility  were  undertaken,  quantifying  mission  consequences,  and  modeling  how  mission 
consequences  depend  on  METOC  outcomes  is  very  difficult.  Mission  consequences  are  multi¬ 
dimensional,  and  often  assessed  subjectively. 

The  role  of  METOC  information  in  helping  commanders  to  make  decisions  that  lead  to  the  best  mission 
results,  an  analyst  must  understand  what  would  happen  if  the  commanders  took  other  courses  of  action  - 
these  consequences,  though  never  observed,  must  be  modeled.  If  decisions  are  determined  in  a  rule-based 
way,  often  certain  combinations  of  decisions  and  METOC  outcomes  are  not  observed  -  therefore  data  are 
not  there  to  collect.24 


24  In  the  analysis  of  platforms  and  weapons  systems,  even  those  that  are  METOC-sensitive,  the  effects  of  METOC 
are  usually  treated  as  threshold-based  limits.  In  testing  and  evaluation,  METOC  impacts  are  usually  not  tested,  at 
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Moreover,  when  commanders  retrospectively  assess  operations,  they  generally  emphasize  contributions  to 
mission  success  over  which  they  had  control  -  and  generally  ascribe  less  importance  to  aspects  like 
METOC  conditions  over  which  they  had  no  control.  Therefore,  even  when  METOC  conditions  affect 
mission  results,  the  impacts  are  rarely  quantified  and  reported. 

All  of  these  challenges  are  especially  difficult  for  modeling  or  collecting  relevant  data  for  modeling  the 
use  and  impact  of  climatology  information.  Climatology-based  decisions  and  their  consequences  are 
separated  in  time  attenuating  both  responsibility  and  causal  relationships. 

METOC  impacts  on  weapon  systems 

I  pursued  public,  private,  and  classified  routes  to  try  to  find  information  on  the  effect  of  METOC 
operations  on  missiles  and  mines.  I  looked  into  the  assumptions  behind  mine-warfare  tactical  decision  aid. 
I  tried  to  identify  academic  experts  in  the  mechanics  of  sea  mines.  At  least  one  expressed  the  belief  that 
high  seas  would  increase  mines’  effective  radius,  and  at  least  one  believed  the  opposite. 

I  tried  common  sources  on  weapons  information  -  such  as  Jane’s,  wikipedia  (which  was  surprisingly 
informative)  -  and  DoD  sources,  such  as  JMEMS  (Joint  Munitions  Effectiveness  Manuals).  I  contacted 
the  Survivability/Vulnerability  Information  Analysis  Center  (SURVIAC),  “the  DoD’s  institution  for 
collecting,  analyzing,  and  disseminating  scientific  and  technical  information  (STI)  related  to  all  aspects  of 
survivability  and  lethality  for  aircraft,  ground  vehicles,  ships  and  spacecraft,  to  conventional  homeland 
security  threats  including  chemical,  biological,  directed  energy,  and  non-lethal  weapons” 
(http://www.bahdavton.com/surviac/).  A  typical  response  is  shown  below. 
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Dr  Regnier 

After  completing  a  search  of  the  SURVIAC  library  I  am  unable  to  give  you  the  information  you  have  requested  The  only  information  that  I  was  able  to  locate  related  to 
your  request  was  in  the  Joint  Munitions  Effectiveness  Manual  which  detailed  each  of  the  threats  designed  targets  effective  ranges  velocity  envelopes  component 
descriptions  as  well  as  reliability  data  The  reliability  data  is  'the  probability  that  the  weapon  system  will  not  malfunction  and  that  the  weapon  will  guide  to  an  impact 
point  within  the  guidance  system  design  cntena.  It  is  not  the  same  as  the  probability  of  hit  or  near  miss '  The  reliability  data  was  simply  a  percentage  not  a  function  of 
range  or  weather  or  target  From  the  data  that  we  have  available  here  it  would  seem  that  any  more  detailed  information  would  require  a  study  performed  for  each  of  the 
weapon  systems 

Because  of  the  challenges  in  measuring,  modeling,  or  otherwise  attempting  to  predict  the  performance 
effects  of  various  METOC  conditions,  the  lack  of  information  I  encountered  for  weapons  systems  may 
also  extend  to  other  systems,  such  as  sensing  equipment,  aircraft  and  seacraft. 

In  at  least  some  cases,  manufacturers  offer  standards  for  METOC  conditions  under  which  their  systems 
may  be  used.  For  example,  this  information  may  be  provided  in  the  NATOPS  (standard  operating 
procedures)  documentation.25  Certainly,  the  DoD  organizations  that  fly  aircraft  have  standards  for  the 
METOC  conditions  that  permit  flying.  However,  these  are  not  based  on  measuring  the  impact  of  varying 
METOC  conditions  on  performance  measures,  such  as  probability  of  successful  takeoff  or  landing.  For 


least  not  at  multiple  levels.  By  contrast,  the  OR/DA  approach  would  involve  analytically  identifying  an  optimal 
threshold,  which  balances  risk  of  adverse  METOC  conditions  with  the  risk  of  failing  to  take  advantage  of  acceptable 
METOC  conditions.  Moreover,  the  optimal  threshold  would  change  as  a  function  of  specific  details  of  a  mission  -  to 
include  how  critical  the  mission  is  in  the  larger  context.  In  practice,  these  thresholds  have  generally  been  identified 
subjectively,  are  fixed  for  all  missions  that  involve  particular  platforms  or  weapons  systems,  and  are  unlikely  to  be 
“optimal”  in  any  given  set  of  circumstances. 

25 1  initially  tried  to  get  a  copy  of  this  document  for  an  F-18,  or  at  least  of  the  relevant  sections,  but  it  is  not  available 
for  free,  and  I  talked  with  some  pilots,  and  am  basing  this  on  their  description  of  the  type  of  information  provided. 
One  route  to  purchase  a  NATOPS  is  http://rareaviation.com/store/catalog 
/product  2801  NATOPS  Flight  Manual  for  Navy  Model  F  A18E  F  Aircraft  cat  888.html. 
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obvious  reasons,  measuring  this  relationship  experimentally  would  be  costly  and  dangerous.  Modeling 
METOC  impacts  on  systems  is  also  likely  to  be  costly  and  difficult  (for  a  modeling  effort)  and  perhaps 
not  worthwhile. 

7.2  Design  of  forecast  and  climatology  systems  to  support  decision-making 

There  is  a  chicken-egg  problem  in  the  design  of  forecast  and  climatology  systems  to  support  operational 
decision-making.  Users  have  neither  decision  processes  nor  models  for  hypothetical  future  systems,  and 
therefore  cannot  inform  design  by  saying  what  they  need.  The  METOC  community  does  not  know  the 
details  of  how  information  may  be  used  and  what  the  operational  impacts  might  be,  and  so  cannot  design 
optimally.  Before  a  hypothetical  METOC  product,  or  product  enhancement  such  as  increased  accuracy, 
becomes  available,  end-users  have  no  incentive  to  identify  ways  to  use  the  product,  let  alone  predict  the 
consequences  of  its  use.  Many  potential  applications  will  not  be  identified  until  the  required  information 
product  is  provided  to  commanders.  Even  once  a  product  is  available,  it  may  take  some  time  for 
commanders  to  gain  familiarity  with  and  confidence  in  the  product  and  begin  to  use  it. 

Figure  5  expands  on  the  influence  diagram  in  Figure  1  by  depicting  in  pink  elements  of  the  METOC 
information  and  decision-making  systems  that  influence  forecast  value  over  the  long  term.  On  the 
METOC  side,  investments  in  observing  systems,  modeling,  and  data  collection  and  storage  affect  the 
quality,  quantity,  lead  time  and  type  of  guidance  available  to  forecasters.  Forecast  methodology  -  the 
processes  by  which  forecasters  integrate  guidance  to  produce  forecasts  -  affects  both  forecast  accuracy 
and  the  format  of  forecast  information.  Examples  of  changes  in  forecast  methodology  include  training  to 
improve  forecaster  judgments,  automation  of  some  portion  of  the  forecast  process,  and  development  of 
new  products  that  may  be  tailored  to  specific  users  or  decisions. 

Improvements  in  any  of  these  elements  should  make  METOC  information  more  accurate,  which  in  turn 
should  affect  how  users  exploit  forecast  (or  climatology).  In  general,  the  more  accurate  the  information, 
the  more  decisions  should  be  influenced  by  it,  and  the  more  value  can  be  extracted  by  using  the 
information  as  a  decision  input.  New  forecast  products  and  improved  product  design  can  also  increase  the 
value  of  METOC  information  by  increasing  the  number  of  decisions  that  can  be  based  on  the  information 
-  for  example,  redesigning  a  threshold-based  product  around  an  operationally  relevant  threshold  could 
increase  the  product’s  value  (see  Section  6.3.2),  even  if  accuracy  does  not  improve. 

This  highlights  the  interaction  between  changes  on  the  METOC  side  and  changes  in  the  decision  context. 
The  development  of  decision  policies  has  been  modeled  in  this  report  as  commanders’  means  of 
exploiting  METOC  information  to  produce  the  best  expected  outcomes.  Decision  policies  may  dictate 
what  information  should  be  included  in  a  decision  process,  as  well  as  what  alternatives  should  be  selected, 
conditional  on  METOC  information.  As  METOC  information  content  and  quality  changes,  so  decision 
policies  should  change.  For  example,  if  tropical  cyclone  track  forecasts  improved  dramatically,  the 
conditions  for  setting  TCCOR  4  might  be  limited  -  perhaps  TCCOR  4  would  be  set  only  when  tropical 
cyclone  formation  potential  was  above  some  threshold. 

As  indicated  in  Figure  5,  over  the  long  run,  commanders  may  also  be  able  to  change  the  alternatives 
available  to  them.  For  example,  in  the  Halsey  scenario,  if  Blue  were  able  to  acquire  missiles  that  require 
different  METOC  conditions  from  the  Type  A  and  B  missiles,  and/or  have  different  costs  and 
effectiveness,  the  decision  policy  would  change  to  reflect  the  new  alternatives,  and  the  value  of  METOC 
information  would  change,  and  possibly  increase. 

The  elements  of  the  METOC  system  represented  in  pink  in  Figure  5  (observation,  modeling,  and 
climatology  data  systems  and  forecast  methodology)  are  often  analyzed  independently  of  the  elements  of 
the  decision  context  (decision  policy  and  available  actions)  and  vice  versa.  However,  their  effects  on 
forecast  value  are  highly  interdependent. 
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Figure  5:  The  system  of  interactions  by  which  forecast  information  influences  consequences. 

There  may  be  a  special  challenge  for  climatology  as  it  may  influence  decisions  more  subtly,  because  clear 
METOC  impacts  on  operations  such  as  limits  presented  by  operational  thresholds  will  not  be  activated 
based  on  climatology.  Decision  and  consequence  models  are  expected  to  be  less  common  for 
climatology-based  decisions.  These  challenges  should  not  discourage  the  METOC  community  from 
considering  the  interdependencies.  The  effects  of  changes  in  METOC  systems  on  forecast  value  may  need 
to  be  considered  qualitatively  and  subjectively.  Neglecting  to  consider  these  interactions  in  developing 
METOC  systems  will  reduce  METOC  value  relative  to  its  potential,  and  perhaps  even  relative  to  its 
current  value. 
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Appendix  A  -  The  Missile  Battle 

The  missile  battle  model  is  based  on  the  basic  salvo  model  by  Hughes  (1995),  pitting  Blue  aircraft  against 
Red  vessels.  Hughes  assumes  that  two  fleets  of  vessels,  each  fleet  consisting  of  one  type  of  vessel  with 
one  type  of  missile,  engage  in  a  missile  exchange. 

The  missile  exchange  is  divided  into  multiple  salvos.  In  each  salvo,  a  fraction  of  Red  vessels  are  disabled 
and  a  fraction  of  Blue  vessels  are  disabled,  as  determined  by  deterministic  formulas  based  on  the  number 
of  “well-aimed  missiles”  fired  by  each  able  vessel,  the  number  of  well-aimed  missiles  required  to  disable 
each  type  of  vessel,  and  the  number  of  missiles  each  vessel  can  defend  against  (per  salvo). 


Table  A-l:  Notation  for  salvo  model,  following  Hughes  (1995) 


R, 

Number  of  able  vessels  in  the  Red  force  at  the  beginning  of  salvo  t 

B, 

Number  of  able  vessels  in  the  Blue  force  at  the  beginning  of  salvo  t 

p(a3) 

Number  of  well-aimed  missiles  fired  by  each  Red  vessel,  a  function  of  the  Red’s  missile 
type  selected  ( a3 )  -  when  METOC  conditions  permit  its  use.  When  METOC  conditions 

do  not  permit  the  use  of  the  missile  type  selected,  p(a3 )  =0. 

P{ai) 

Number  of  well-aimed  missiles  fired  by  each  Blue  aircraft,  a  function  of  the  Blue’s 
missile  type  selected  (a2)~  when  METOC  conditions  permit  its  use.  When  METOC 

conditions  do  not  permit  the  use  of  the  missile  type  selected,  (3  (a., )  =0. 

ri 

Number  of  hits  by  Blue’s  missiles  required  to  disable  one  Red  vessel  (Red’s  staying 
power) 

by 

Number  of  hits  by  Red’s  missiles  required  to  disable  one  Blue  aircraft  (Blue’s  staying 
power) 

Number  of  well-aimed  missiles  destroyed  by  each  Red  vessel  (Red’s  defensive  capability) 

bi 

Number  of  well-aimed  missiles  destroyed  by  each  Blue  aircraft  (Blue’s  defensive 
capability) 

A R, 

Number  of  Red  vessels  (potentially)  destroyed  by  Blue’s  missiles  during  salvo  t  ;  (unless 

A R>R,) 

M, 

Number  of  Blue  aircraft  (potentially)  destroyed  by  Red’s  missiles  during  salvo  t  ;  (unless 

A B>B,) 

In  addition  to  values  for  parameters  described  above,  the  following  assumptions  have  been  added  to  the 
Hughes  model: 

•  Each  missiles  type  is  either  usable  or  unusable,  according  to  METOC  conditions,  as  detailed  in 
Table  3  in  the  body  of  the  report.  If  missiles  are  usable,  all  fired  missiles  are  well-aimed. 

•  Red  has  two  vessel  types  -  escorts  and  landing  craft,  and  Blue  does  not  distinguish  between  them 
in  targeting.  Parameters  such  as  number  of  missiles  fired  per  salvo  per  vessel  are  weighted 
averages  based  on  the  fraction  of  escort  vessels  among  total  Red  vessels,  and  assuming  no 
armament  on  the  landing  vessels. 


One  salvo  corresponds  to  one  fighter  sortie.  Up  to  six  sorties  occur  between  the  time  Red  vessels 
cross  the  neutral  line  and  the  time  they  land.  (Using  the  parameter  values  given  in  Table  A-2, 
either  the  Red  or  Blue  fleet  will  be  completely  destroyed  within  six  salvos.) 


For  each  salvo. 


,  ,  j3(a2)B  - kR 

Rl+i  =  max  |0,  Rt  —  A Rr  ] ,  A Rt  = - - —  and  similarly 


K\  =  max{0,£,-A£r},  A Bt  = 


p{a  3)R,-b3B, 


The  notation  is  defined  in  Table  A-l  and  parameter  values  used  in  the  Flalsey  scenario  example  and  the 
extended  Flalsey  scenario  (with  climatology)  are  given  in  Table  A-2. 


Table  A-2:  Parameters  of  salvo  model 
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p(<h) 

[ 0.55  if  a3  =  C  (Type  C  missile  used) 

[0.27  if  o3  =  D  (Type  D  missile  used) 

P{a2) 

[10  if  a2  =  A  (Type  A  missile  used) 

[  6  if  a2  =  B  (Type  B  missile  used) 

ri 

4.27 

bi 

2 

0 

h 

1 
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Appendix  B  -  Modeling  and  Solving  Multi-stage  Problems 


The  Decision  Tree 

Multi-stage  problems  are  often  represented  using  a  decision  tree,  which  makes  it  easy  to  understand 
precedence  relationships  and  the  analysis  of  the  decisions.  Events  include  decisions,  receipt  of 
information,  and  resolution  of  uncertainty.  In  a  decision  tree,  events  proceed  chronologically  from  left  to 
right.  An  example  of  a  partial  decision  tree  for  the  extended  Halsey  scenario  is  given  in  Figure  B-l. 

Red 

missile 

selection 


Figure  B-l:  Decision  tree  for  expanded  Halsey  scenario 


Nodes  represent  possible  states  of  the  system  -  for  every  possible  scenario  (defined  by  information  sets 
received,  choices  of  alternatives,  and  outcomes  of  early  events)  preceding  each  stage  in  the  model,  there 
will  be  a  separate  node.  The  unique  arrow  entering  each  node  indicates  which  path  (state  of  system)  leads 
to  the  node.  By  convention,  the  node’s  shape  and  color  indicate  whether  it  is  a  decision  (blue  square),  an 
uncertain  variable  (red  circle).  Uncertainty  nodes  are  called  chance  nodes.  In  the  case  of  decision  nodes, 
arrows  leaving  the  node  indicate  the  alternatives  available  given  that  state  of  the  system.  In  the  case  of 
chance  nodes,  arrows  leaving  the  node  indicate  the  possible  outcomes  of  the  uncertain  event,  usually  with 
their  probabilities  indicated,  as  in  Figure  B-2. 

The  consequences  of  each  possible  path  through  the  tree  are  shown  at  the  end  (far  right)  of  the  decision 
tree. 
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Analyzing  the  multi-stage  problem 

As  discussed  in  Section  4.4,  solutions  are  reached  by  backward  induction:  start  with  the  last  event(s)  in 
the  sequence,  and  work  backwards.  During  the  solution  process,  each  node  is  assigned  a  value  as  follows: 

•  At  each  decision  node,  the  commander  chooses  the  branch  that  optimizes  expected  consequences 
(minimizes  total  cost)  -  this  is  the  optimal  decision  for  that  node.  Eliminate  branches  that  are  not 
optimal  (prune  the  tree).  Each  decision  node  is  assigned  the  value  associated  with  the  remaining 
branch.  Eliminated  branches  should  not  figure  in  earlier  decisions  -  assuming  the  commander 
will  always  make  the  right  decisions,  the  eliminated  branches  will  not  be  selected. 

•  Every  chance  node  takes  the  expected  value  of  the  nodes  to  its  right,  where  expectation  is  taken 
conditional  on  all  information  to  that  point. 

This  process  is  illustrated  in  Figure  B-l  and  Figure  B-2.  In  Figure  B-l,  the  top  node  at  the  far  right  is 
considered  -  this  represents  a  decision  node  for  Red  (hence  the  mixed  coloring  of  the  square).  This  node 
represents  the  scenario  in  which  climatology  is  =  1 ,  at  the  first  decision  opportunity,  ax  =  K  (Kadena) 

is  selected,  w,  =  1  occurs  and  becomes  known  to  the  commander,  forecast  (p2  =  1  is  issued,  missile  type 

a2=  A  is  selected,  and  finally  (after  Blue’s  missile  selection  decision)  w,  =  1  occurs  and  becomes  known 

to  the  Red  commander.  Red  has  two  choices  -  choosing  missile  Type  C  will  result  in  72  Red  landing 
vessels  surviving,  while  choosing  Type  D  will  result  in  zero  Red  landing  vessels  surviving.  In  this 
situation,  we  assume  Red  will  rationally  choose  Type  C  missiles.  The  other  alternative  is  eliminated  and 
its  branch  is  pruned  from  the  tree  (note  that  it  does  not  appear  in  Figure  B-2).  The  value  72  is  assigned  to 
the  decision  node. 

In  Figure  B-2,  all  the  right-most  decision  nodes  have  been  analyzed.  Next,  two  of  the  288  chance  nodes 
representing  the  outcome  of  METOC  conditions  in  the  engagement  zone  ( w2 )  are  analyzed  -  by  taking 

the  expectation  over  the  four  possible  outcomes.  The  node  values  are  indicated  in  pink.  Next,  moving  to 
the  left,  one  of  the  decision  nodes  for  Blue’s  missile-selection  decision  is  analyzed  -  Blue  will  choose  the 
branch  with  the  lower  value  (minimizing  the  expected  number  of  Red  landing  craft  surviving  to  land),  and 
the  other  branch  is  eliminated. 

The  process  continues,  with  the  tree  rolled  back  and  pruned  at  each  decision  node,  until  the  first  decision 
can  be  analyzed. 
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Figure  B-2:  Partially  pruned  decision  tree  for  expanded  Halsey  scenario 


Appendix  C  -  Sea  Mine  Effectiveness 

Like  the  missile  exchange  (described  in  Appendix  A)  sea  mining  is  modeled  very  simply.  We  assume  12 
mines  per  bomber  aircraft,  and  10  bombers,  so  each  bomber  can  drop  120  mines  per  sortie.  The  transit 
time  for  Red  is  18  hours,  unless  weather  is  very  adverse  ( w2  =4).  Therefore,  if  the  bombers  are  stationed 

in  Alaska  (a,  =  3 ),  there  will  be  zero  sorties  completed  unless  weather  conditions  are  very  severe 

O,  =4). 


Table  C-l:  Sea  mine  model  notation 


frontage 

In  feet,  the  length  of  the  landing  beach 

troops 

=  328 1  •  >  approximately  1  km  per  800-person  battalion 

troops 

Troop  capacity  of  all  Red  landing  craft  =  17,580 

ru 

Triggering  radius  of  mine 

M 

Number  of  mines  laid  before  Red  vessels  reach  landing  beach 

nRVD 

Number  of  mines  required  to  disable  a  Red  vessel 

A 

Average  number  of  mines  in  fraction  of  minefield  “swept”  by  a  given  vessel 
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- M - M 

frontage 


Further  assumptions  are: 

•  Mines  are  randomly  (uniformly)  distributed  in  minefield 

•  The  number  of  mines  is  much  greater  than  the  number  that  are  triggered  (  i.e.  probability  of  a  given 
ship  encountering  a  mine  does  not  decrease  as  mines  are  triggered  by  the  ship  or  other  ships) 

•  Radius  for  triggering  is  constant  (assumed  30  feet) 

•  Number  of  mines  triggered  by  a  ship  □  Poisson  1) . 


Number  of  mines  that  can  be  triggered  by  a  vessel  and  have  the  vessel  survive  is  nRVD  - 1 ,  while  if 
nRVD  are  triggered,  the  vessel  is  disabled;  nRVD  =2  assumed. 

Fraction  of  vessels  disabled  =  expected  probability  that  the  number  of  mines  triggered  by  a  ship 
exceeds  nRVD  - 1 


probability  that 
m  mines  are 
triggered 


probability  that 
m  mines  are 
triggered 
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